Need a sql efficient query - mysql

Question

I have around 6 million rows in the table and I am query the table with below query.
SELECT * FROM FD_CPC_HISTORICAL_DATA WHERE id IN (SELECT MAX(id) FROM FD_CPC_HISTORICAL_DATA WHERE fb_ads_account_id=1462257067274960 AND created_at BETWEEN '2019-12-13 00:00:00' AND '2019-12-13 23:59:59' GROUP BY source_text) \G
I have created index for fb_ads_account_id, created_at, source_text. id is primary key.
My question is why this query takes around 9 seconds to get the result even though I have created indexes?
Is there any other way to create this query more efficient?
Here is mysql explain command explanation

Answer 1

This is your query:
SELECT hd.*
FROM FD_CPC_HISTORICAL_DATA hd
WHERE hd.id IN (SELECT MAX(hd2.id)
FROM FD_CPC_HISTORICAL_DATA hd2
WHERE hd2.fb_ads_account_id = 1462257067274960 AND
hd2.created_at >= '2019-12-13' AND
hd2.created_at < '2019-12-14'
GROUP BY source_text
);
I would recommend writing this as:
SELECT hd.*
FROM FD_CPC_HISTORICAL_DATA hd
WHERE hd.fb_ads_account_id = 1462257067274960 AND
hd.id = (SELECT MAX(hd2.id)
FROM FD_CPC_HISTORICAL_DATA hd2
WHERE hd2.fb_ads_account_id = hd.hd.fb_ads_account_id AND
hd2.source_text = hd.source_tx AND
hd2.created_at >= '2019-12-13' AND
hd2.created_at < '2019-12-14'
);
For this query, you want an index on FD_CPC_HISTORICAL_DATA(fb_ads_account_id, source_text,created_at).

Answer 2

This query probably can be performed without a subquery against the same table ie:
SELECT * FROM FD_CPC_HISTORICAL_DATA
WHERE fb_ads_account_id=1462257067274960
AND created_at BETWEEN '2019-12-13 00:00:00' AND '2019-12-13 23:59:59'
ORDER BY id DESC LIMIT 1
if you want the max ID. Or something similar, I am not sure you need the GROUP BY to get the desired result.

Answer 3

I think the index is exactly what you need. The part in the EXPLAIN that confuses me is the (guesstimated?) amount of rows from the subquery being so different from the one in the primary query.
To be honest, I'm not very familiar with MYSQL, but in MSSQL I would give it a try to first dump the results from the subquery into a temporary table, put a unique clustered index on it and then select everything from the original table joined to said temporary table on the ID column. (Don't use IN, use JOIN as there can't be any doubles in the temporary table)
This might also show where all the time is being spent.
My guess is that this is mostly a statistics issue but I don't really know how to force an update of the statistics on the index in MYSQL.
(there is some talk about FLUSH TABLE in https://dzone.com/articles/updating-innodb-table-statistics-manually but it seems to come with some downsides too, use with care)

Answer 4

SELECT f.*
FROM
( SELECT source_text, MAX(created_at) AS mx
FROM FD_CPC_HISTORICAL_DATA
WHERE fb_ads_account_id=1462257067274960
AND created_at >= '2019-12-13'
AND created_at < '2019-12-13' + INTERVAL 1 DAY
GROUP BY source_text
) AS x
JOIN FD_CPC_HISTORICAL_DATA AS f
ON f.account_id = x.account_id
AND f.source_text = x.source_text
AND f.created_at = x.mx
Then you need this composite index:
INDEX(account_id, source_text, created_at) -- in this order
If this does not quite work because of duplicate entries with the same created_at, then a tweak may be possible.