MySQL InnoDB - GROUP BY on many items - mysql

Question

I've got a table that has id, date, ad_id, ad_network, ad_event columns.
In my database there are millions of distinct ad_id each has a few events associated with them.
When I try to use GROUP BY on the ad_id to count each event it takes so long there is 503 error.
I need to count distinct AdClickThru and AdImpression so that I can calculate the CTR.
The problem is that one user can click many times, so I must count only one AdClickThru.
The query is below:
SELECT
`ad_network`,
`ad_id`,
SUM(DISTINCT CASE WHEN `ad_event` = "AdImpression" THEN 1 ELSE 0 END) as AdImpression,
SUM(DISTINCT CASE WHEN `ad_event` = "AdClickThru" THEN 1 ELSE 0 END) as AdClickThru
FROM `ads`
WHERE 1
AND `ad_event` IN ("AdImpression", "AdClickThru")
AND SUBSTR(`date`, 1, 7) = "2020-08"
GROUP BY `ad_id`
I have indexes on ad_id and ad_event + date but it does not help much.
How can I optimize this query?
The database will grow to billions of entries and more.
#edit
Forgot to mention that the code above is inner part of outer query:
SELECT
`ad_network`,
SUM(`AdImpression`) as cnt_AdImpression,
SUM(`AdClickThru`) as cnt_AdClickThru,
100 * SUM(`AdClickThru`) / SUM(`AdImpression`) as ctr
FROM (
SELECT
`ad_network`,
`ad_id`,
SUM(DISTINCT CASE WHEN `ad_event` = "AdImpression" THEN 1 ELSE 0 END) as AdImpression,
SUM(DISTINCT CASE WHEN `ad_event` = "AdClickThru" THEN 1 ELSE 0 END) as AdClickThru
FROM `ads`
WHERE 1
AND `ad_event` IN ("AdImpression", "AdClickThru")
AND SUBSTR(`date`, 1, 7) = "2020-08" -- better performance
GROUP BY `ad_id`
) a
GROUP BY `ad_network`
ORDER BY ctr DESC

Answer 1

The problem is that one user can click many times, so I must count only one AdClickThru.
Then use MAX(), not COUNT(DISTINCT). This gives the same result as your expression, and is much more efficient. I would also recommend rewriting the date filter so it is index-friendly:
SELECT
`ad_network`,
`ad_id`,
MAX(`ad_event` = 'AdImpression') as AdImpression,
MAX(`ad_event` = 'AdClickThru') as AdClickThru
FROM `ads`
WHERE 1
AND `ad_event` IN ('AdImpression', 'AdClickThru')
AND `date` >= '2020-08-01'
AND `date` < '2020-09-01'
GROUP BY `ad_id`
Notes:
the presence of ad_network in the select clause is hitching me: if there are several values per ad_id, it is undefined which will be picked. Either put this column in the group by clause as well, or use an aggregate function in the sélect clause (such as MAX(ad_network) - or if you are ok with an arbitrary value, then be explicit about it with any_value()
use single quotes for literal strings rather than double quotes (this is the SQL standard)

Answer 2

There is no need for 2 separate aggregations in the main query and the subquery.
You want to count the distinct ad_ids for each of the 2 cases:
SELECT ad_network,
COUNT(DISTINCT CASE WHEN ad_event = 'AdImpression' THEN ad_id END) AS cnt_AdImpression,
COUNT(DISTINCT CASE WHEN ad_event = 'AdClickThru' THEN ad_id END) AS cnt_AdClickThru,
100 *
COUNT(DISTINCT CASE WHEN ad_event = 'AdClickThru' THEN ad_id END) /
COUNT(DISTINCT CASE WHEN ad_event = 'AdImpression' THEN ad_id END) AS ctr
FROM ads
WHERE ad_event IN ('AdImpression', 'AdClickThru') AND SUBSTR(date, 1, 7) = '2020-08'
GROUP BY ad_network
ORDER BY ctr DESC
The problem here is that you have to repeat the expressions for cnt_AdImpression and cnt_AdClickThru.
You can calculate these expressions in a subquery:
SELECT ad_network, cnt_AdImpression, cnt_AdClickThru,
100 * cnt_AdClickThru / cnt_AdImpression AS ctr
FROM (
SELECT ad_network,
COUNT(DISTINCT CASE WHEN ad_event = 'AdImpression' THEN ad_id END) AS cnt_AdImpression,
COUNT(DISTINCT CASE WHEN ad_event = 'AdClickThru' THEN ad_id END) AS cnt_AdClickThru
FROM ads
WHERE ad_event IN ('AdImpression', 'AdClickThru') AND SUBSTR(date, 1, 7) = '2020-08'
GROUP BY ad_network
) t
ORDER BY ctr DESC