How to speed up Group by query - mysql

Question

I have a mysql query that take 30 seconds to run. there are more than 3millions rows in the table
here is the db structure :
text (VARCHAR(64)),
kpi1 (INT),
kpi2 (INT),
position (DECIMAL),
date(DATE)
device (VARCHAR(32))
Here is the query :
select date, sum(kpi1), sum(kpi2) FROM `table_name` GROUP BY date ;
Explain method gives me this result :
ID | select type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtred | extra
1 | SIMPLE | table_name | NULL | index | UNIQUE,DATE | DATE | 3 | NULL | 3316480 | 100.00 | NULL
I have an index on date.
Here the result with profiling :
mysql> show profile for query 1;
+----------------------+-----------+
| Status | Duration |
+----------------------+-----------+
| starting | 0.000080 |
| checking permissions | 0.000011 |
| Opening tables | 0.000021 |
| init | 0.000023 |
| System lock | 0.000011 |
| optimizing | 0.000007 |
| statistics | 0.000021 |
| preparing | 0.000019 |
| Sorting result | 0.000007 |
| executing | 0.000005 |
| Sending data | 32.814836 |
| end | 0.000011 |
| query end | 0.000009 |
| closing tables | 0.000009 |
| freeing items | 0.000082 |
| cleaning up | 0.000013 |
+----------------------+-----------+
16 rows in set, 1 warning (0,00 sec)
Any idea ?

Answer 1

If the data on historical dates is static (as in, not changing because the date / activity is already done), then this is a perfect example of when to use a summary table. Create a separate table that is nothing but the date and the aggregates as you need them. Do that for all days prior to the current, so only at the end of the day, you insert (such as some daily trigger) the sum of the prior day. You could even include the count of records, something like
insert into MyDailySummaryTable
( Date, kpi1Sum, kpi2Sum, numRecs )
select date,
sum(kpi1) kpi1Sum,
sum(kpi2) kpi2Sum,
count(*) numRecs
FROM
`table_name`
where
date < curdate()
GROUP BY
date ;
then for each day after the initial load
insert into MyDailySummaryTable
( Date, kpi1Sum, kpi2Sum, numRecs )
select date,
sum(kpi1) kpi1Sum,
sum(kpi2) kpi2Sum,
count(*) numRecs
FROM
`table_name`
where
date = date_add( curdate(), interval -1 day )
GROUP BY
date ;
If your "date" field has timestamp information too, you may need to adjust the query to ignore the time portions.
Then, when trying to run your totals, you can just query from the MyDailySummaryTable directly and have instant results.
You could even expand the query aggregate table to include the counts per device in case you ever wanted to find tracking info for that one specific device too.