MySQL specific query performance tuning - mysql

Question

I have troubles with MySQL query performance.
Table (InnoDB):
+--------------------+---------------------+------+-----+-------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+---------------------+------+-----+-------------------+-------+
| st_resource_id | varchar(32) | NO | MUL | NULL | |
| st_sub_resource_id | varchar(32) | YES | | NULL | |
| st_title | varchar(500) | YES | | NULL | |
| st_resource_type | varchar(100) | NO | MUL | NULL | |
| st_site_id | tinyint(4) | NO | MUL | NULL | |
| st_time | timestamp | NO | MUL | CURRENT_TIMESTAMP | |
| st_user_id | int(10) unsigned | YES | | NULL | |
| st_full_access | tinyint(1) unsigned | YES | | NULL | |
+--------------------+---------------------+------+-----+-------------------+-------+
Indexes:
+---------------+------------+------------------+--------------+--------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------------+------------+------------------+--------------+--------------------+-----------+-------------+----------+--------+------+------------+---------+
| nr_statistics | 1 | resource_id | 1 | st_resource_id | A | 1546165 | NULL | NULL | | BTREE | |
| nr_statistics | 1 | resource_id | 2 | st_sub_resource_id | A | 1546165 | NULL | NULL | YES | BTREE | |
| nr_statistics | 1 | st_time | 1 | st_time | A | 1546165 | NULL | NULL | | BTREE | |
| nr_statistics | 1 | st_site_id | 1 | st_site_id | A | 16 | NULL | NULL | | BTREE | |
| nr_statistics | 1 | st_resource_type | 1 | st_resource_type | A | 16 | 10 | NULL | | BTREE | |
+---------------+------------+------------------+--------------+--------------------+-----------+-------------+----------+--------+------+------------+---------+
Query:
SELECT st_resource_id AS docId, count(*) AS cnt
FROM nr_statistics
WHERE
st_resource_type = 'document'
AND st_sub_resource_id = 'text'
AND st_time > DATE_SUB(NOW(), INTERVAL 7 DAY)
AND st_site_id = 1
GROUP BY st_resource_id
ORDER BY cnt DESC
LIMIT 0, 5;
Query plan:
+----+-------------+---------------+-------+-------------------------------------+-------------+---------+------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------+-------+-------------------------------------+-------------+---------+------+---------+----------------------------------------------+
| 1 | SIMPLE | nr_statistics | index | st_time,st_site_id,st_resource_type | resource_id | 197 | NULL | 1581044 | Using where; Using temporary; Using filesort |
+----+-------------+---------------+-------+-------------------------------------+-------------+---------+------+---------+----------------------------------------------+
Table has ~1,666,383 rows. The query runs extremely slow. In MySQL process list I see this query in "copy to tmp table phase" for a long time (> 1 minute). Query generates heavy I/O load. I can't understand what to do to fix the problem and speed-up query execution.
If the problem is a result of wrong indexes, so what indexes will be right?
UPD. I created new composite index:
| nr_statistics | 1 | st_site_id_2 | 1 | st_site_id | A | 16 | NULL | NULL | | BTREE | |
| nr_statistics | 1 | st_site_id_2 | 2 | st_resource_type | A | 16 | NULL | NULL | | BTREE | |
| nr_statistics | 1 | st_site_id_2 | 3 | st_sub_resource_id | A | 752018 | NULL | NULL | YES | BTREE | |
| nr_statistics | 1 | st_site_id_2 | 4 | st_time | A | 1504037 | NULL | NULL | | BTREE | |
| nr_statistics | 1 | st_site_id_2 | 5 | st_resource_id | A | 1504037 | NULL | NULL | | BTREE | |
Now query plan is:
+----+-------------+---------------+-------+---------------+--------------+---------+------+-------+-----------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------+-------+---------------+--------------+---------+------+-------+-----------------------------------------------------------+
| 1 | SIMPLE | nr_statistics | range | st_site_id_2 | st_site_id_2 | 406 | NULL | 21168 | Using where; Using index; Using temporary; Using filesort |
+----+-------------+---------------+-------+---------------+--------------+---------+------+-------+-----------------------------------------------------------+
The query now runs very fast (as 0.0x sec), but I have to force using new index:
SELECT st_resource_id as docId, count( * ) AS Cnt
FROM nr_statistics
USE INDEX (st_site_id_2)
WHERE st_resource_type = 'document'
AND st_sub_resource_id = 'text'
AND st_time > DATE_SUB( NOW( ) , INTERVAL 7 DAY )
AND st_site_id = 1
GROUP BY st_resource_id
ORDER BY cnt DESC
LIMIT 0 , 5;
While the problem is resolved (not beautiful but effective way), I still have some open questions (see comments).

Answer 1

Create a composite index on (st_site_id, st_resource_type, st_sub_resourse_id, st_time, st_resource_id).
However, you will still have temporary and filesort in the plan because you are ordering on COUNT(*) which is not indexable.
If you need to run this query fast and often, you would have to create an aggregate table which would store counts for each site/resource/subresourse/week combination and update it in a trigger.

Answer 2

Did you try creating a composite index on st_resource_type, st_resource_id, st_time and st_site_id? It looks to me like you have several indexes, but most are on a single column, or maybe 2 columns. By having a composite index with more of the columns you need, it may improve the performance.

Answer 3

When doing queries with multiple where clauses the order in which you write them should match the order in which you wrote your query.
In your particular case it would be:
CREATE INDEX stats_index ON nr_statistics (st_resource_type, st_sub_resource_id, st_time, st_site_id);
This should give you a pretty good speed boost.