I am using Excel 365.
I have a Power Query (PQ) that reads off a MS SQL database and I need to set parameters for that query from a table in the same Excel workbook.
I created the parameters in PQ, when I feed them manually the query works
But when I link those parameters to my Excel table get the following error message
Formula.Firewall: Query 'SQL_Query' (step 'Source') references other queries or steps, so it may not directly access a data source. Please rebuild this data combination.
I changed the security settings but it did not work.
I've seen several posts that work but in instances where both data sources come directly only from Excel.
let
Source = Sql.Database("www.com", "db", [Query="
DECLARE #START_DATE AS DATE
DECLARE #END_DATE AS DATE
DECLARE #HYP AS VARCHAR(4)
SET #START_DATE = convert(datetime, " & Start_Dte & ",101)
SET #END_DATE = convert(datetime, " & End_Dte & ",101)
SET #HYP = " & Hyp & " -- = Query_Parameters('ParameterTable', 'Store')
select
case WHEN view1.STORE_NAME <> 'NULL' then view1.STORE_NAME ELSE view2.STORE end as [Store Name],
case WHEN view1.Store_Street <> 'NULL' then view1.Store_Street ELSE view2.Store_Street end as [Store Street],
case WHEN view1.Store_City <> 'NULL' then view1.Store_City ELSE view2.Store_City end as [Store City],
case WHEN view1.Store_State <> 'NULL' then view1.Store_State ELSE view2.[Store State] end as [Store State],
case WHEN view1.Store_zip <> 'NULL' then view1.Store_zip ELSE view2.Store_ZIP end as [Store ZIP],
case WHEN view1.Store <> 'NULL' then view1.Store ELSE view2.Store end as [Store],
case WHEN view1.ZIP_CODE <> 'NULL' then view1.ZIP_CODE ELSE view2.ZIP_CODE end as [ZIP Code],
case WHEN view2.Distance ='[1-5 MILE]' then '[0-5 miles]' ELSE
case WHEN view2.Distance ='[6-10 MILE]' then '[06-10 miles]' ELSE
case WHEN view2.Distance ='[+50 MILE]' then '[50+ miles]' ELSE
case WHEN view1.Distance <> 'NULL' then replace(view1.Distance,'MILES','miles') ELSE replace(view2.Distance,'MILES','miles')
end end
end
end as Distance,
case WHEN view1.Provider <> 'NULL' then view1.Provider ELSE view2.Provider end as Provider,
isnull(view1.Location,'New / Old') as Location,
case WHEN view1.[Lead Source] <> 'NULL' then view1.[Lead Source] ELSE '' end as [Lead Source],
isnull(view1.[YEAR],'') as [Year],
isnull(view1.[month],'') as [Month],
isnull(view1.[Opps],'') as [Opps],
isnull(view1.[Compass_Sales],'') as [Compass Sales],
case WHEN view2.TC_Terrain='Yes' THEN 'Yes' ELSE 'No' END as [TC Terrain]
from (
----Opps & Sales by Provider and by ZIP ---------
SELECT
b.STORE_NAME,
f.STREET1 as Store_Street,
g.CITY as Store_City,
g.STATE as Store_State,
g.ZIP_CODE as Store_ZIP,
b.STORE_STORE_ID as Store,
a.CONTACT_ZIP as ZIP_CODE,
a.DISTANCE as [Distance],
a.media_name as [Provider],
d.Location_name as [Location],
ccategory_name as [Lead Source],
YEAR(TRANSACTION_DATE) as [YEAR],
MONTH(TRANSACTION_DATE) as [MONTH],
SUM(Opportunities) as Opps,
SUM(Compass_Sales) as Compass_Sales
FROM [DB].[fact].[FACT_SALES_TRAFFIC_DETAIL] a
left join db.dim.dim_store b with(nolock) on b.store_id = a.store_id
left join db.dim.dim_lead_category c with(nolock)on a.SOURCE = ccategory
left join db.dim.dim_Location d with(nolock)on a.Location_Key = d.Location_key
left join db.dim.dim_market e with(nolock)on b.market_id = e.market_id
left join DB.DIM.DIM_STORE f with(nolock) on b.STORE_STORE_ID = f.STORE_STORE_ID
left join db.dim.dim_zipcode g with(nolock) on f.ZIP = g.ZIP_CODE
WHERE b.store_Store_id in (#HYP)
and a.Source = 'e'
and TRANSACTION_DATE between #START_DATE and #END_DATE
and a.MEDIA_NAME='TC'
GROUP BY
b.STORE_NAME,
b.STORE_STORE_ID,
d.Location_name,
a.media_Name,
ccategory_name,
YEAR(TRANSACTION_DATE),
MONTH(TRANSACTION_DATE),
CONTACT_ZIP,
a.DISTANCE,
f.STREET1,
g.CITY,
g.ZIP_CODE,
g.STATE) as view1 full outer join (
---True Car ZIP code Terrain------
SELECT
b.STORE_name as STORE,
b.STORE_STORE_ID as [Store],
b.STREET1 as [Store_Street],
c.CITY as [Store_City],
c.STATE as [Store State],
b.ZIP as [Store_ZIP],
INCLUDED_ZIP AS ZIP_CODE,
DISTANCE AS Distance,
'TC' AS Provider,
'Yes' as TC_Terrain
FROM BITESTDB.TIMLINM.TC_ZIPCODE a
left join DB.DIM.DIM_STORE b with(nolock) on b.STORE_id = a.STORE_id
left join db.dim.dim_zipcode c with(nolock) on c.ZIP_CODE= b.ZIP
WHERE STORE_STORE_ID = #HYP AND INCLUDED_ZIP <> '') as view2
on view1.ZIP_CODE = view2.zip_code
ORDER BY Distance
", MultiSubnetFailover=true])
in
Source
The easy way: open Power Query, then File > Options > CURRENT FILE > Privacy > Ignore
I have two table in my database main_item_stock and stock_history
I want to create a report to show that how many stock in, stock out and wastage for each item in main_item_stock, for every action that user do will insert a record into stock_history.
main_item_stock
-----------------------------------------------
stkitm_id stkitm_code stkitm_name stkitm_qty
-----------------------------------------------
1 S001 Apple 10.000
2 S002 Orange 5.000
-----------------------------------------------
stock_history
-------------------------------------------------------------------------------------
stkhis_id sktitm_id stkitm_code stkitm_name stkhis_type stkhis_qty created_date
1 1 S001 Apple Wastage 1.000 2017/03/13
2 2 S002 Orange Stock Out 0.500 2017/03/12
3 2 S002 Orange Stock In 0.100 2017/03/13
-------------------------------------------------------------------------------------
Result that I want:-
----------------------------------------------------------------------------------------------
stkitm_code stkitm_name created_date item_onhand item_stockin item_stockout item_wastage
----------------------------------------------------------------------------------------------
S001 Apple 2017/03/13 9.000 0 0 1
S002 Orange 2017/03/12 4.500 0 0.500 0
S002 Orange 2017/03/13 4.600 0.100 0 0
*item_onhand = main_item_stock.stkitm_qty
I have tried below query but stockin,stockout,wastage always return same value
query = "SELECT main_item_stock.stkitm_code,main_item_stock.stkitm_name,main_item_stock.stkitm_qty,
(SELECT SUM(stkhis_qty) FROM stock_history WHERE stkhis_type = 'Stock Out'"
If dtp_from.Checked Then
query &= " AND DATE(stock_history.created_date) >= '" & Format(CDate(dtp_from.Text), "yyyy-MM-dd") & "'
AND DATE(stock_history.created_date) <= '" & Format(CDate(dtp_to.Text), "yyyy-MM-dd") & "'"
End If
query &= ") AS 'item_stockout',
(SELECT SUM(stkhis_qty) FROM stock_history WHERE stkhis_type = 'Wastage'"
If dtp_from.Checked Then
query &= " AND DATE(stock_history.created_date) >= '" & Format(CDate(dtp_from.Text), "yyyy-MM-dd") & "'
AND DATE(stock_history.created_date) <= '" & Format(CDate(dtp_to.Text), "yyyy-MM-dd") & "'"
End If
query &= ") AS 'item_wastage',
(SELECT SUM(stkhis_qty) FROM stock_history WHERE stkhis_type = 'Stock In'"
If dtp_from.Checked Then
query &= " AND DATE(stock_history.created_date) >= '" & Format(CDate(dtp_from.Text), "yyyy-MM-dd") & "'
AND DATE(stock_history.created_date) <= '" & Format(CDate(dtp_to.Text), "yyyy-MM-dd") & "'"
End If
query &= ") AS 'item_stockin'
FROM main_item_stock
LEFT JOIN stock_history
ON main_item_stock.stkitm_id = stock_history.stkitm_id"
query &= " GROUP BY stock_history.stkitm_id"
How can I change the query and get the result that I want?
UPDATE:
I have tried another query, this time the return data is correct but can I use this table to create the report that I want?
SELECT SUM(stkhis_qty),stkitm_name,stkhis_type,DATE(created_date)
FROM stock_history
WHERE DATE(created_date) >= '2017/03/12'
GROUP BY stkitm_code,stkhis_type,DATE(created_date)
result:-
How can I changed image1 table to image2? The report need to have all these column
UPDATE:
This is main_item_stock
This is stock_history
Result should look like picture below, both type also can.
*the last record date on the picture is wrong should be 3/17/2017
This should get you pretty close...
*edited to reflect the change of items on hand rather than the current total number of items on hand.
SELECT
sh.stkitm_code,
sh.stkitm_name,
SUM(IF(stkhis_type = 'Stock In',
sh.stkhis_qty,
-1 * sh.stkhis_qty)) AS change_of_item_onhand,
SUM(IF(stkhis_type = 'Stock In',
sh.stkhis_qty,
0)) AS item_stockin,
SUM(IF(stkhis_type = 'Stock Out',
sh.stkhis_qty,
0)) AS item_stockout,
SUM(IF(stkhis_type = 'Wastage',
sh.stkhis_qty,
0)) AS item_wastage,
sh.created_date
FROM stock_history sh
LEFT JOIN main_item_stock mis ON mis.stkitm_code = sh.stkitm_code
WHERE sh.created_date > "<Oldest date>"
AND sh.created_date < "<Youngest date>"
GROUP BY stkitm_code
Which should result in output that looks like:
Although you'd probably want to group by the stkitm_code and date.
Alternatively, removing the "SUM"s and "GROUP BY" will itemize it, but the item_onhand column won't be cumulative (there might be a slick way to do that though)
SELECT
sh.stkitm_code,
sh.stkitm_name,
IF(stkhis_type = 'Wastage' OR
stkhis_type = 'Stock Out',
mis.stkitm_qty - sh.stkhis_qty,
mis.stkitm_qty + sh.stkhis_qty) AS item_onhand,
IF(stkhis_type = 'Stock In',
sh.stkhis_qty,
0) AS item_stockin,
IF(stkhis_type = 'Stock Out',
sh.stkhis_qty,
0) AS item_stockout,
IF(stkhis_type = 'Wastage',
sh.stkhis_qty,
0) AS item_wastage,
sh.created_date
FROM stock_history sh
LEFT JOIN main_item_stock mis ON mis.stkitm_code = sh.stkitm_code
WHERE sh.created_date > "<Oldest date>"
AND sh.created_date < "<Youngest date>"
Which will end up looking like this:
Hope this helps!
I've written an SSRS report to help me keep track of SSRS subscriptions. I've repurposed a script that will use Reportserver.dbo.Subscriptions.LastStatus to view email recipients, however, it will only list the first 520 characters of LastStatus. Because some of our distribution lists are quite large, some of the names that my script searches for are not being found (even though they are part of the distribution). Below is the script that I am using:
SELECT Reportname = c.Name
,FileLocation = c.Path
,SubscriptionDesc=su.Description
,Subscriptiontype=su.EventType
,su.LastStatus
,su.LastRunTime
,Schedulename=sch.Name
,ScheduleType = sch.EventType
,ScheduleFrequency =
CASE sch.RecurrenceType
WHEN 1 THEN 'Once'
WHEN 2 THEN 'Hourly'
WHEN 4 THEN 'Daily/Weekly'
WHEN 5 THEN 'Monthly'
END
,su.Parameters
FROM Reportserver.dbo.Subscriptions su
JOIN Reportserver.dbo.Catalog c
ON su.Report_OID = c.ItemID
JOIN Reportserver.dbo.ReportSchedule rsc
ON rsc.ReportID = c.ItemID
AND rsc.SubscriptionID = su.SubscriptionID
JOIN Reportserver.dbo.Schedule Sch
ON rsc.ScheduleID = sch.ScheduleID
WHERE LastStatus like #Email
ORDER BY LastRunTime DESC
Any code that I have found online uses the LastStatus column to display this data. If anyone has any suggestions as to a more complete way for me to list all of the members of the report distribution list, I would appreciate it.
Below is SQL to query for the full text of the subscription parameters. I think this will work with extremely long address lists, but I don't have a test server with long address lists available right now.
If using this in production, I'd probably throw in a couple of WITH ( NOLOCK )'s and wouldn't expect support from MS on problems.
;
WITH subscriptionXmL
AS (
SELECT
SubscriptionID ,
OwnerID ,
Report_OID ,
Locale ,
InactiveFlags ,
ExtensionSettings ,
CONVERT(XML, ExtensionSettings) AS ExtensionSettingsXML ,
ModifiedByID ,
ModifiedDate ,
Description ,
LastStatus ,
EventType ,
MatchData ,
LastRunTime ,
Parameters ,
DeliveryExtension ,
Version
FROM
ReportServer.dbo.Subscriptions
),
-- Get the settings as pairs
SettingsCTE
AS (
SELECT
SubscriptionID ,
ExtensionSettings ,
-- include other fields if you need them.
ISNULL(Settings.value('(./*:Name/text())[1]', 'nvarchar(1024)'),
'Value') AS SettingName ,
Settings.value('(./*:Value/text())[1]', 'nvarchar(max)') AS SettingValue
FROM
subscriptionXmL
CROSS APPLY subscriptionXmL.ExtensionSettingsXML.nodes('//*:ParameterValue') Queries ( Settings )
)
SELECT
*
FROM
SettingsCTE
WHERE
settingName IN ( 'TO', 'CC', 'BCC' )
I also found this query from SQL Server MSDN Social;
Original Query Author MSDN Profile: Sandip Shinde
SELECT
c.Name AS ReportName,
'Next Run Date' = CASE next_run_date
WHEN 0 THEN null
ELSE
substring(convert(varchar(15),next_run_date),1,4) + '/' +
substring(convert(varchar(15),next_run_date),5,2) + '/' +
substring(convert(varchar(15),next_run_date),7,2)
END,
'Next Run Time' = isnull(CASE len(next_run_time)
WHEN 3 THEN cast('00:0'
+ Left(right(next_run_time,3),1)
+':' + right(next_run_time,2) as char (8))
WHEN 4 THEN cast('00:'
+ Left(right(next_run_time,4),2)
+':' + right(next_run_time,2) as char (8))
WHEN 5 THEN cast('0' + Left(right(next_run_time,5),1)
+':' + Left(right(next_run_time,4),2)
+':' + right(next_run_time,2) as char (8))
WHEN 6 THEN cast(Left(right(next_run_time,6),2)
+':' + Left(right(next_run_time,4),2)
+':' + right(next_run_time,2) as char (8))
END,'NA'),
Convert(XML,[ExtensionSettings]).value('(//ParameterValue/Value[../Name="TO"])[1]','nvarchar(50)') as [To]
,Convert(XML,[ExtensionSettings]).value('(//ParameterValue/Value[../Name="CC"])[1]','nvarchar(50)') as [CC]
,Convert(XML,[ExtensionSettings]).value('(//ParameterValue/Value[../Name="RenderFormat"])[1]','nvarchar(50)') as [Render Format]
,Convert(XML,[ExtensionSettings]).value('(//ParameterValue/Value[../Name="Subject"])[1]','nvarchar(50)') as [Subject]
---Example report parameters: StartDateMacro, EndDateMacro & Currency.
,Convert(XML,[Parameters]).value('(//ParameterValue/Value[../Name="StartDateMacro"])[1]','nvarchar(50)') as [Start Date]
,Convert(XML,[Parameters]).value('(//ParameterValue/Value[../Name="EndDateMacro"])[1]','nvarchar(50)') as [End Date]
,Convert(XML,[Parameters]).value('(//ParameterValue/Value[../Name="Currency"])[1]','nvarchar(50)') as [Currency]
,[LastStatus]
,[EventType]
,[LastRunTime]
,[DeliveryExtension]
,[Version]
FROM
dbo.[Catalog] c
INNER JOIN dbo.[Subscriptions] S ON c.ItemID = S.Report_OID
INNER JOIN dbo.ReportSchedule R ON S.SubscriptionID = R.SubscriptionID
INNER JOIN msdb.dbo.sysjobs J ON Convert(nvarchar(128),R.ScheduleID) = J.name
INNER JOIN msdb.dbo.sysjobschedules JS ON J.job_id = JS.job_id
According to MS (https://learn.microsoft.com/en-us/sql/reporting-services/subscriptions/manage-subscription-owners-and-run-subscription-powershell?view=sql-server-ver16), you can also use this powershell command:
$webSRV = New-WebServiceProxy -Uri "http://myservername/ReportServer/ReportService2010.asmx" -Namespace SSRS.ReportingService2010 -UseDefaultCredential;
$mySubsList = $webSRV.ListSubscriptions("/");
$mySubsList | select *
I was trying to get multiple counts of occuring fields from 2 or more tables and I solved that issue from this question,
Nested queries to get count with two conditions
The solution worked but when tried in the following data structure,
It executes but never displays any result. but with the previous question i posted this kind of solution worked fine..Can anyone please help on this issue..it doesn't even say there is a syntax error.
EDIT: sql query
SELECT t1.timeStamp, t1.localIp, t2.localPort, t3.localGeo, t4.isp, t5.foreignIp, t6.foreignPort, t7.foreignGeo, t8.infection,t1.timeStampCount, t1.localIpCount, t2.localPortCount, t3.localGeoCount, t4.ispCount, t5.foreignIpCount, t6.foreignPortCount, t7.foreignGeoCount, t8.infectionCount
FROM
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(timeStamp) AS 'timeStampCount',COUNT(localIp) AS 'localIpCount'
FROM (
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'
FROM tbl_shadowserver_bot_geo
UNION ALL
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'
FROM tbl_www_cymru_com
) c
GROUP BY timeStamp,localIp)t1
JOIN
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(localPort) AS 'localPortCount'
FROM (
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'
FROM tbl_shadowserver_bot_geo
UNION ALL
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'
FROM tbl_www_cymru_com
) c
GROUP BY timeStamp,localIp,localPort)t2
ON t1.timeStamp=t2.timeStamp
JOIN
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(localGeo) AS 'localGeoCount'
FROM (
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'
FROM tbl_shadowserver_bot_geo
UNION ALL
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'
FROM tbl_www_cymru_com
) c
GROUP BY timeStamp,localIp,localPort,localGeo)t3
ON t1.timeStamp=t3.timeStamp
JOIN
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(isp) AS 'ispCount'
FROM (
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'
FROM tbl_shadowserver_bot_geo
UNION ALL
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'
FROM tbl_www_cymru_com
) c
GROUP BY timeStamp,localIp,localPort,localGeo,isp)t4
ON t1.timeStamp=t4.timeStamp
JOIN
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(foreignIp) AS 'foreignIpCount'
FROM (
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'
FROM tbl_shadowserver_bot_geo
UNION ALL
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'
FROM tbl_www_cymru_com
) c
GROUP BY timeStamp,localIp,localPort,localGeo,isp,foreignIp)t5
ON t1.timeStamp=t5.timeStamp
JOIN
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(foreignPort) AS 'foreignPortCount'
FROM (
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'
FROM tbl_shadowserver_bot_geo
UNION ALL
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'
FROM tbl_www_cymru_com
) c
GROUP BY timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort)t6
ON t1.timeStamp=t6.timeStamp
JOIN
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(foreignGeo) AS 'foreignGeoCount'
FROM (
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'
FROM tbl_shadowserver_bot_geo
UNION ALL
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'
FROM tbl_www_cymru_com
) c
GROUP BY timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo)t7
ON t1.timeStamp=t7.timeStamp JOIN
(SELECT timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection,COUNT(infection) AS 'infectionCount'
FROM (
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', port AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', "" AS 'infection'
FROM tbl_shadowserver_bot_geo
UNION ALL
SELECT date_format(timestamp, '%Y-%m-%d %h') AS 'timeStamp', ip AS 'localIp', "" AS 'localPort', "" AS 'localGeo', "" AS 'isp', "" AS 'foreignIp', "" AS 'foreignPort', "" AS 'foreignGeo', Formatreport AS 'infection'
FROM tbl_www_cymru_com
) c
GROUP BY timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection)t8
ON t1.timeStamp=t8.timeStamp ORDER BY timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection
I'll first try to restate your question in my own words, to make it clear what this answer here does. For each record in the union of two tables, you want one row in the result set. That row should contain additional information: for a given set of selected columns you want to count the rows from the input which match the current rows in all selected columns. In each step you add to the set of selected rows. So first you count all rows with the same time stamp. Next you count all the rows with the same time stamp and the same IP address. And so on.
As you repeatedly select from the same union of tables, it might be prudent to introduce a short name for it. You can do so by creating a VIEW. Next, you want to use that view as the first factor of your join. This gives you one row of output for every row of input. For the added statistics, you join to that a subquery which counts rows, grouping by all the selected columns for that count. You want to join that subquery to the rest of your query using all these selected columns. So for example:
CREATE VIEW view_both_tables AS
SELECT date_format(timestamp, '%Y-%m-%d %h') AS `timeStamp`, ip AS localIp, …
FROM tbl_shadowserver_bot_geo
UNION ALL
SELECT date_format(timestamp, '%Y-%m-%d %h') AS `timeStamp`, ip AS localIp, …
FROM tbl_www_cymru_com;
SELECT t0.*,
t1.countSameTime,
t2.countSameTimeAndLocalIp,
t3.countSameTimeLocalIpAndInfection
FROM view_both_tables t0
LEFT JOIN (
SELECT `timeStamp`,
COUNT(*) AS `countSameTime`
FROM view_both_tables
GROUP BY `timeStamp`
) t1 USING (`timeStamp`)
LEFT JOIN (
SELECT `timeStamp`, localIp,
COUNT(*) AS `countSameTimeAndLocalIp`
FROM view_both_tables
GROUP BY `timeStamp`, localIp
) t2 USING (`timeStamp`, localIp)
LEFT JOIN (
SELECT `timeStamp`, localIp, infection,
COUNT(*) AS `countSameTimeLocalIpAndInfection`
FROM view_both_tables
GROUP BY `timeStamp`, localIp, infection
) t3 USING (`timeStamp`, localIp, infection)
;
This query was taken from my SQL fiddle. It only covers up to three selected columns, as the other columns didn't contain any data in your example, and as the idea should be clear from this.
In general, it is more efficient to COUNT(*) instead of counting one particular column. The only exceptions are when the column you want to count may contain NULL values which should be omitted from the count, or when you're using COUNT(DISTINCT …). But as my interpretation of your question referred to counting records, not rows, neither of these exceptions applies.
As a further note on style, using single quotation marks to quote column names is rather confusing SQL style. Usually, strings are delimited using single quotation marks, and columns uisng backticks. What you wrote is valid syntax for the default MySQL configuration, but confusing nonetheless.
If performance is an issue, you might try executing a single query using GROUP BY … WITH ROLLUP. Storing the result to a temporary table, you could derive the result you asked for from repeatedly joining that table to itself. No sure whether there would be any performance gain, but it might be worth a try.