Create 3rd Table SQL with difference of 2 other tables - mysql

Afternoon all
I am trying to create a view with 3 tables.
Create view breakdown as
SELECT x.month_
, x.returns_
, y.sales_
, z.profit_
FROM
(SELECT SUM(Return_Value_Eur) AS returns_
, monthname(Return_Date) AS month_
FROM returns
GROUP BY month_) x
INNER JOIN
(SELECT SUM(Total_Price_Eur) AS sales_
, monthname(Order_Date) AS month_
FROM orders
GROUP BY month_) y ON x.month_ = y.month_
INNER JOIN
(SELECT (SUM(Total_Price_Eur)-SUM(Return_Value_Eur)) AS profit_
, monthname(Order_Date) AS month_
FROM returns, orders
GROUP BY month_) z ON y.month_ = z.month_
I have the following code however I'm struggling with the profit table. This table should be a difference between the y.sales and x.returns. However the figures are coming up as follows, the profit column should be showing 394.
Month_ Returns_ Sales_ Profit_
January 108 502 -251
Any help would be greatly appreciated.

Don't mix the old-school comma syntax for the join operation with the JOIN keyword; actually, just ditch the old-school comma syntax altogether for new queries.
Looks like the problem is the cartesian product of the join between returns and orders, the inline view returning "profit".
It's not clear what we are trying to achieve (broken SQL is like that, in that it fails to accurately communicate the actual specification)
It looks to me (and this is just a guess here) is that the intent is to subtract total returns from total sales, by month.
I'd do it like this:
SELECT v.month_
, IFNULL( SUM(v.returns_) ,0) AS returns_
, IFNULL( SUM(v.sales_ ) ,0) AS sales
, IFNULL( SUM(v.sales_ ) ,0)
- IFNULL( SUM(v.returns_) ,0) AS net_
FROM (
SELECT 0.0 AS sales_
, SUM(r.return_value_eur) AS returns_
, MONTHNAME(r.return_date) AS month_
, MONTH(r.return_date) AS month_num
FROM returns r
GROUP
BY MONTHNAME(r.return_date)
, MONTH(r.return_date)
UNION ALL
SELECT SUM(o.total_price_eur) AS sales_
, 0.0 AS returns_
, MONTHNAME(o.order_date) AS month_
, MONTH(o.order_date) AS month_num
FROM orders o
GROUP
BY MONTHNAME(o.order_date)
, MONTH(o.order_date)
) v
GROUP
BY v.month_
, v.month_num
ORDER
BY v.month_num
Note that this query ignores the year, which is a bit odd, mashing together November 2019, with November 2018, 2017, et al.
The example query orders rows by calendar order, January, February, March, ... that's the reason for the additional month_num column.
If there are no rows in returns or orders for a given month, then that month will not appear in the resultset.
To guarantee rows returned for all 12 calendar months, I would use row source that was guaranteed to return me one row for each month. Then do outer joins to aggregated returns by month and aggregated sales by month. (I'd also have the queries work with the integer value for the month, and then translate the integer to a string "month name" as a final output step.)
Something like this:
SELECT MONTHNAME(STR_TO_DATE(c.month_num,'%m')) AS month_
, IFNULL( tr.returns_ ) ,0) AS returns_
, IFNULL( ts.sales_ ) ,0) AS sales_
, IFNULL( ts.sales_ ) ,0)
- IFNULL( tr.returns_ ) ,0) AS net_
FROM ( -- row source for calendar month 1..12
SELECT 1 AS month_num UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4
UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8
UNION ALL SELECT 9 UNION ALL SELECT 10 UNION ALL SELECT 11 UNION ALL SELECT 12
) c
LEFT
JOIN ( -- total returns by month
SELECT SUM(r.return_value_eur) AS returns_
, MONTH(r.return_date) AS month_num
FROM returns r
GROUP
BY MONTH(r.return_date)
) tr
ON tr.month_num = c.month_num
LEFT
JOIN ( -- total orders by month
SELECT SUM(o.total_price_eur) AS sales_
, MONTH(o.order_date) AS month_num
FROM orders o
GROUP
BY MONTH(o.order_date)
) ts
ON ts.month_num = c.month_num
ORDER
BY c.month_num

Related

MySQL. Select without 1 day

Base: name, rate, dt.
Every day cost is inserted, I need to know the maximum cost without the first day. The first day of all products is different.
With all dates
SELECT `name`, MAX(rate) AS max
FROM `base`
GROUP BY `name`
This is similar to what I want, but not working.
SELECT `name`, MAX(rate) AS max, MIN(dt) AS min_dt
FROM `base`
WHERE `dt` > `min_dt`
GROUP BY `name`
Eample base
skirt, 6, 2018-10-10 00:00:00
skirt, 7, 2018-10-11 00:00:00
cap, 7, 2018-10-11 00:00:00
skirt, 8, 2018-10-12 00:00:00
cap, 6, 2018-10-12 00:00:00
Need
skirt, 8
cap, 6
One approach is to use an inline view to get the min_dt for each name. Then we can join and exclude the minimum date rows
Something like this:
SELECT b.name
, MAX(b.rate) AS `max`
FROM ( SELECT d.name
, MIN(d.dt) AS min_dt
FROM `base` d
GROUP
BY d.name
) m
JOIN `base` b
ON b.dt > m.min_dt
AND b.name = m.name
GROUP
BY b.name
There are other query patterns that will achieve an equivalent result. My preference would to avoid a correlated subquery, but something like this would also return the specified result:
SELECT b.name
, MAX(b.rate) AS `max`
FROM `base` b
WHERE b.dt > ( SELECT MIN(d.dt)
FROM `base` d
WHERE d.name = b.name
)
GROUP
BY b.name
(With both of these query forms, if there is only row in base for a given name, the query will not return a row for that name.)
You should try to exclude the min dt using subqueries:
SELECT name, MAX(rate) AS max
FROM (SELECT name, rate, dt
FROM base B
WHERE dt NOT IN (SELECT MIN(dt) FROM base WHERE name=B.name )) as A
GROUP BY name
Fiddle here

MySql - Selecting MAX & MIN and returning the corresponding rows

I trying to get the last 6 months of the min and max of prices in my table and display them as a group by months. My query is not returning the corresponding rows values, such as the date time for when the max price was or min..
I want to select the min & max prices and the date time they both occurred and the rest of the data for that row...
(the reason why i have concat for report_term, as i need to print this with the dataset when displaying results. e.g. February 2018 -> ...., January 2018 -> ...)
SELECT metal_price_id, CONCAT(MONTHNAME(metal_price_datetime), ' ', YEAR(metal_price_datetime)) AS report_term, max(metal_price) as highest_gold_price, metal_price_datetime FROM metal_prices_v2
WHERE metal_id = 1
AND DATEDIFF(NOW(), metal_price_datetime) BETWEEN 0 AND 180
GROUP BY report_term
ORDER BY metal_price_datetime DESC
I have made an example, extract from my DB:
http://sqlfiddle.com/#!9/617bcb2/4/0
My desired result would be to see the min and max prices grouped by month, date of min, date of max.. and all in the last 6 months.
thanks
UPDATE.
The below code works, but it returns back rows from beyond the 180 days specified. I have just checked, and it is because it joining by the price which may be duplicated a number of times during the years.... see: http://sqlfiddle.com/#!9/5f501b/1
You could use twice inner join on the subselect for min and max
select a.metal_price_datetime
, t1.highest_gold_price
, t1.report_term
, t2.lowest_gold_price
,t2.metal_price_datetime
from metal_prices_v2 a
inner join (
SELECT CONCAT(MONTHNAME(metal_price_datetime), ' ', YEAR(metal_price_datetime)) AS report_term
, max(metal_price) as highest_gold_price
from metal_prices_v2
WHERE metal_id = 1
AND DATEDIFF(NOW(), metal_price_datetime) BETWEEN 0 AND 180
GROUP BY report_term
) t1 on t1.highest_gold_price = a.metal_price
inner join (
select a.metal_price_datetime
, t.lowest_gold_price
, t.report_term
from metal_prices_v2 a
inner join (
SELECT CONCAT(MONTHNAME(metal_price_datetime), ' ', YEAR(metal_price_datetime)) AS report_term
, min(metal_price) as lowest_gold_price
from metal_prices_v2
WHERE metal_id = 1
AND DATEDIFF(NOW(), metal_price_datetime) BETWEEN 0 AND 180
GROUP BY report_term
) t on t.lowest_gold_price = a.metal_price
) t2 on t2.report_term = t1.report_term
simplified version of what you should do so you can learn the working process.
You need calculate the min() max() of the periods you need. That is your first brick on this building.
you have tableA, you calculate min() lets call it R1
SELECT group_field, min() as min_value
FROM TableA
GROUP BY group_field
same for max() call it R2
SELECT group_field, max() as max_value
FROM TableA
GROUP BY group_field
Now you need to bring all the data from original fields so you join each result with your original table
We call those T1 and T2:
SELECT tableA.group_field, tableA.value, tableA.date
FROM tableA
JOIN ( ... .. ) as R1
ON tableA.group_field = R1.group_field
AND tableA.value = R1.min_value
SELECT tableA.group_field, tableA.value, tableA.date
FROM tableA
JOIN ( ... .. ) as R2
ON tableA.group_field = R2.group_field
AND tableA.value = R2.max_value
Now we join T1 and T2.
SELECT *
FROM ( .... ) as T1
JOIN ( .... ) as T2
ON t1.group_field = t2.group_field
So the idea is if you can do a brick, you do the next one. Then you also can add filters like last 6 months or something else you need.
In this case the group_field is the CONCAT() value

SQL Queries to analyse Employee Database

I am looking for queries, using which I can analyze a general employee database. This is for Data Analysis.
Tried this for monthly employee trend
SELECT
dt.FullDateAlternateKey as 'Date'
, count(1) as ActiveCount
FROM DimDate dt
LEFT JOIN (SELECT 'Active' as 'EmpStatus', * FROM DimEmployee) emp
-- regular active employees
ON (dt.FullDateAlternateKey between emp.StartDate and ISNULL(emp.EndDate,'9999-12-31'))
WHERE
dt.FullDateAlternateKey = EOMONTH(dt.FullDateAlternateKey)
GROUP BY
dt.FullDateAlternateKey
ORDER BY
1;
also found CTE use for finding employee hierarchy
WITH DirectReports (ManagerID, EmployeeID, Title, DeptID, Level)
AS
(
-- Anchor member definition
SELECT e.ParentEmployeeKey, e.EmployeeKey, e.Title, e.DepartmentName,
0 AS Level
FROM DimEmployee AS e
WHERE e.ParentEmployeeKey IS NULL
UNION ALL
-- Recursive member definition
SELECT e.ParentEmployeeKey, e.EmployeeKey, e.Title, e.DepartmentName,
Level + 1
FROM DimEmployee AS e
INNER JOIN DirectReports AS d
ON e.ParentEmployeeKey = d.EmployeeID
)
-- Statement that executes the CTE
SELECT ManagerID, EmployeeID, Title, DeptID, Level
FROM DirectReports
WHERE DeptID = 'Information Services' OR Level = 0
also, some good queries to analyze the sales data
-- Show each sales average for Group, Country, and Region all in one query
SELECT DISTINCT
t.SalesTerritoryGroup
, t.SalesTerritoryCountry
, t.SalesTerritoryRegion
, AVG(s.SalesAmount) OVER(PARTITION BY t.SalesTerritoryGroup ) as 'GroupAvgSales'
, AVG(s.SalesAmount) OVER(PARTITION BY t.SalesTerritoryCountry ) as 'CountryAvgSales'
, AVG(s.SalesAmount) OVER(PARTITION BY t.SalesTerritoryRegion ) as 'RegionAvgSales'
FROM FactInternetSales s
JOIN DimSalesTerritory t ON
s.SalesTerritoryKey = t.SalesTerritoryKey
WHERE
YEAR(s.OrderDate) = 2013
ORDER BY
1,2,3
Use additional aggregations to understand more about product sales such as the distribution of sales etc..
SELECT
cat.EnglishProductCategoryName 'Category'
, sub.EnglishProductSubcategoryName 'SubCategory'
, count(1) 'Count' -- How many sales where there?
, sum(s.SalesAmount) 'Sales' -- How much sales did we have?
, avg(s.SalesAmount) 'Avg_SalesAmount' -- What was the Avg sale amount?
, min(s.SalesAmount) 'Min_SaleAmount' -- What was the Min sale amount?
, max(s.SalesAmount) 'Max_SaleAmount' -- What was the Max sale amount
FROM FactInternetSales s
LEFT JOIN DimProduct p ON s.ProductKey = p.ProductKey
LEFT JOIN DimProductSubcategory sub ON p.ProductSubcategoryKey = sub.ProductSubcategoryKey
LEFT JOIN DimProductCategory cat ON sub.ProductCategoryKey = cat.ProductCategoryKey
-- must use group by in order for aggregation to work properly
GROUP BY
cat.EnglishProductCategoryName -- column aliases aren't allowed
, sub.EnglishProductSubcategoryName
ORDER BY
cat.EnglishProductCategoryName
, sub.EnglishProductSubcategoryName
-- Calculate the customer acquisition funnel
SELECT
c.FirstName
, c.LastName
, c.DateFirstPurchase
, DATEDIFF(d,c.DateFirstPurchase,getdate()) as 'DaysSinceFirstPurchase' -- How long have they been a customer?
FROM DimCustomer c
ORDER BY 3 DESC
-- Calculate a Monthly average of customer tenure
SELECT
EOMONTH(c.DateFirstPurchase) as 'MonthOfFirstPurchase' -- What month did they become a customer?
, DATEDIFF(d,EOMONTH(c.DateFirstPurchase),getdate()) as 'DaysSinceFirstPurchase' -- How long have they been a customer?
, COUNT(1) as 'CustomerCount' -- How manY customers are there for this month?
FROM DimCustomer c
GROUP BY EOMONTH(c.DateFirstPurchase)
ORDER BY 2 DESC
-- Show the top product Sub Categories for each year
SELECT
count(DISTINCT s.SalesOrderNumber) 'OrderCount' -- use 1 instead of a field for faster performance
, RANK() OVER (PARTITION BY YEAR(s.OrderDate) ORDER BY sum(s.SalesAmount) DESC) 'SalesRank'
, sum(s.SalesAmount) 'TotalSales'
, cat.EnglishProductCategoryName 'Category'
, sub.EnglishProductSubcategoryName 'SubCategory'
, YEAR(s.OrderDate) 'Year'
FROM FactInternetSales s
INNER JOIN DimProduct p ON s.ProductKey = p.ProductKey
INNER JOIN DimProductSubcategory sub ON p.ProductSubcategoryKey = sub.ProductSubcategoryKey
INNER JOIN DimProductCategory cat ON sub.ProductCategoryKey = cat.ProductCategoryKey
-- must use group by in order for aggregation to work properly
GROUP BY
cat.EnglishProductCategoryName -- column aliases aren't allowed
, sub.EnglishProductSubcategoryName
, YEAR(s.OrderDate)
ORDER BY YEAR(s.OrderDate), SUM(s.SalesAmount) DESC;
-- first, create weekly sales totals
SELECT SUM(s.SalesAmount) 'WeeklySales'
, DATEPART(ww, s.OrderDate) as 'WeekNum'
FROM FactInternetSales s
WHERE YEAR(s.OrderDate) = 2013
GROUP BY
DATEPART(ww, s.OrderDate)
ORDER BY
DATEPART(ww, s.OrderDate) ASC
-- use that subquery as our source and calculate the moving average
SELECT
AVG(WeeklySales) OVER (ORDER BY WeekNum ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) as AvgSales
, WeeklySales as 'TotalSales'
, WeekNum
FROM (
SELECT SUM(s.SalesAmount) 'WeeklySales'
, DATEPART(ww, s.OrderDate) as 'WeekNum'
FROM FactInternetSales s
WHERE YEAR(s.OrderDate) = 2013
GROUP BY
DATEPART(ww, s.OrderDate)
) AS s
GROUP BY
WeekNum, WeeklySales
ORDER BY
WeekNum ASC
-- Running Total
SELECT
SUM(MonthlySales) OVER (PARTITION BY SalesYear ORDER BY SalesMonth ROWS UNBOUNDED PRECEDING) as YTDSales
, MonthlySales as 'MonthlySales'
, SalesYear
, SalesMonth
FROM (
SELECT SUM(s.SalesAmount) 'MonthlySales'
, MONTH(s.OrderDate) as 'SalesMonth'
, year(s.OrderDate) as 'SalesYear'
FROM FactInternetSales s
GROUP BY
MONTH(s.OrderDate)
, year(s.OrderDate)
) AS s
GROUP BY
SalesMonth, SalesYear, MonthlySales
ORDER BY
SalesYear, SalesMonth ASC
-- Get Prev Year Sales
WITH MonthlySales (YearNum, MonthNum, Sales)
AS
(
SELECT d.CalendarYear, d.MonthNumberOfYear, SUM(s.SalesAmount)
FROM DimDate d
JOIN FactInternetSales s ON d.DateKey = s.OrderDateKey
GROUP BY d.CalendarYear, d.MonthNumberOfYear
)
-- Get Current Year and join to CTE for previous year
SELECT
d.CalendarYear
, d.MonthNumberOfYear
, ms.Sales PrevSales
, SUM(s.SalesAmount) CurrentSales
FROM DimDate d
JOIN FactInternetSales s ON d.DateKey = s.OrderDateKey
JOIN MonthlySales ms ON
d.CalendarYear-1 = ms.YearNum AND
d.MonthNumberOfYear = ms.MonthNum
GROUP BY
d.CalendarYear
, d.MonthNumberOfYear
, ms.Sales
ORDER BY
1 DESC, 2 DESC
-- Now calculate the % change Year over Year
WITH MonthlySales (YearNum, MonthNum, Sales)
AS
(
SELECT d.CalendarYear, d.MonthNumberOfYear, SUM(s.SalesAmount)
FROM DimDate d
JOIN FactInternetSales s ON d.DateKey = s.OrderDateKey
GROUP BY d.CalendarYear, d.MonthNumberOfYear
)
-- Get Current Year and join to CTE for previous year
SELECT
d.CalendarYear
, d.MonthNumberOfYear
, ms.Sales PrevSales
, SUM(s.SalesAmount) CurrentSales
, (SUM(s.SalesAmount) - ms.Sales) / SUM(s.SalesAmount) 'PctGrowth'
FROM DimDate d
JOIN FactInternetSales s ON d.DateKey = s.OrderDateKey
JOIN MonthlySales ms ON
d.CalendarYear-1 = ms.YearNum AND
d.MonthNumberOfYear = ms.MonthNum
GROUP BY
d.CalendarYear
, d.MonthNumberOfYear
, ms.Sales
ORDER BY
1 DESC, 2 DESC

Query in MySQL that get values from two tables and put them in a specific order

I want to get values from this two tables and put them in a GridView in a specific way.
First table is estimated income and the second is actual income.
This is the estimated income table:
and this is the actual income:
I want to get values from two tables and display them in a GridView to look like this table:
The code that i stuck in is:
select t as which , SUM(Amount_income_table) as Actual,
SUM(estimated_amount) as Estimated
from ((select MONTHNAME(Date_income_table) as T,
select SUM(Amount_income_table) as Actual
FROM bacci.income_table WHERE year(Date_income_table)='2017' GROUP BY
monthname(date_income_table))
)union all
(SELECT SUM(estimated_amount) as Estimated
FROM bacci.estimated_income_table WHERE estimated_year='2017' GROUP BY
estimated_month) order by MONTHNAME(Date_income_table);
Assuming that bacci.estimated_income_table is guaranteed to contain rows for all twelve months we want to return, and assuming estimated_month column is integer value 1 thru 12, I'd do something like this:
EDIT
It appears that estimated_month is actually the month name January, February, ... so we derive the month number (1 thru 12) from that ...
SELECT d.y_ AS `year`
, d.mn_ AS `month`
, IFNULL(SUM(d.actual),0) AS `actual`
, IFNULL(SUM(d.estimated),0) AS `estimated`
FROM (
SELECT YEAR(it.date_income_table) AS `y_`
, MONTH(it.date_income_table) AS `m_`
, MONTHNAME(it.date_income_table) AS `mn_`
, SUM(it.amount_income_table) AS `actual`
, NULL AS `estimated`
FROM bacci.income_table it
WHERE it.date_income_table >= '2017-01-01'
AND it.date_income_table < '2017-01-01' + INTERVAL 1 YEAR
GROUP
BY YEAR(it.date_income_table)
, MONTH(it.date_income_table)
, MONTHNAME(it.date_income_table)
UNION ALL
SELECT ei.estimated_year AS `y_`
, MONTH(STR_TO_DATE(ei.estimated_month,'%M')) AS `m_`
, ei.estimated_month AS `mn_`
, NULL AS `actual`
, SUM(ei.estimated_amount) AS `estimated`
FROM bacci.estimated_income_table ei
WHERE ei.estimated_year = '2017'
GROUP
BY ei.estimated_year
, MONTH(STR_TO_DATE(ei.estimated_month,'%M'))
, ei.estimated_month
) d
GROUP
BY d.y_
, d.m_
, d.mn_
ORDER
BY d.y_
, d.m_

MySQL Join Two Queries Horizontally

I have a query that works correctly to pull a series of targets and total hours worked for company A. I would like to run the exact same query for company B and join them on a common date, which happens to be grouped by week. My current query:
SELECT * FROM (
SELECT org, date,
( SELECT SUM( target ) FROM target WHERE org = "companyA" ) AS companyA_target,
SUM( hours ) AS companyA_actual
FROM time_management_system
WHERE org = "companyA"
GROUP BY WEEK( date )
ORDER BY DATE
) q1
LEFT JOIN (
SELECT org, date,
( SELECT SUM( target ) FROM target WHERE org = "companyB" ) AS companyB_target,
SUM( hours ) AS companyB_actual
FROM time_management_system
WHERE org = "companyB"
GROUP BY WEEK( date )
ORDER BY DATE
) q2
ON q1.date = q2.date
The results show all of the dates / information of companyA, however companyB only shows sporadic data. Separately, the two queries will show the exact same set of dates, just with different information in the 'target' and 'actual' columns.
companyA 2012-01-28 105.00 39.00 NULL NULL NULL NULL
companyA 2012-02-05 105.00 15.00 NULL NULL NULL NULL
companyA 2012-02-13 105.00 60.50 companyB 2012-02-13 97.50 117.50
Any idea why I'm not getting all the information for companyB?
As a side note, would anybody be able to point in the direction of converting each row's week value into a column? With companyA and companyB as the only two rows?
I appreciate all the help! Thanks.
WITH no date apparent in the target table, the summation will be constant across all weeks. So, I have performed a pre-query for only those "org" values of company A and B with a group by. This will ensure only 1 record per "org" so you don't get a Cartesian result.
Then, I am querying the time_management_system ONCE for BOTH companies. Within the field computations, I am applying an IF() to test the company value and apply when correct. The WEEK activity is the same for both in the final result, so I don't have to do separately and join. This also prevents the need of having the date column appear twice. I also don't need to explicitly add the org column names as the final column names reflect that.
SELECT
WEEK( tms.date ) as GrpWeek,
IF( tms.org = "companyA", TargetSum.CompTarget, 00000.00 )) as CompanyATarget,
SUM( IF( tms.org = "companyA", tms.hours, 0000.00 )) as CompanyAHours,
IF( tms.org = "companyB", TargetSum.CompTarget, 00000.00 )) as CompanyBTarget,
SUM( IF( tms.org = "companyB", tms.hours, 000.00 )) as CompanyBHours
from
Time_Management_System tms
JOIN ( select
t.org,
SUM( t.target ) as CompTarget
from
Target T
where
t.org in ( "companyA", "companyB" )
group by
t.org ) as TargetSums
ON tms.org = TargetSums.org
where
tms.org in ( "companyA", "companyB" )
group by
WEEK( tms.date )
order by
WEEK( tms.date )
Both of your subqueries are wrong.
Either you want this:
SELECT
org,
WEEK(date),
( SELECT SUM( target ) FROM target WHERE org = "companyB" ) AS companyB_target,
SUM( hours ) AS companyB_actual
FROM time_management_system
WHERE org = "companyB"
GROUP BY WEEK( date )
Or else you want this:
SELECT
org,
date,
( SELECT SUM( target ) FROM target WHERE org = "companyB" ) AS companyB_target,
SUM( hours ) AS companyB_actual
FROM time_management_system
WHERE org = "companyB"
GROUP BY date
The way you are doing it now is not correctly formed SQL. In pretty much any other database your query would fail immediately with an error. MySQL is more lax and runs the query but gives indeterminate results.
GROUP BY and HAVING with Hidden Columns