Why does MySQL not use indexes for this select query? - mysql

Question

Why does this query not use indexes here? The table uses the InnoDB engine.
explain SELECT null as id,
-> up_time,
-> reg_date,
-> refer,
-> MAX(IFNULL(visits_count,0)) as visits_count,
-> MAX(IFNULL(register_count,0)) as register_count,
-> MAX(IFNULL(players_count,0)) as players_count,
-> MAX(IFNULL(activity_count,0)) as activity_count,
-> MAX(IFNULL(payment_users_count,0)) as payment_users_count,
-> MAX(IFNULL(payment_count,0)) as payment_count,
-> MAX(IFNULL(payment_sum,0)) as payment_sum FROM stats_refers
->
-> WHERE
->
-> stats_refers.reg_date < 1435006800
-> AND stats_refers.up_time < 1435006800
->
-> GROUP BY stats_refers.refer, stats_refers.reg_date;
And the explain:
+----+-------------+--------------+------+----------------------------+------+---------+------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------+------+----------------------------+------+---------+------+---------+----------------------------------------------+
| 1 | SIMPLE | stats_refers | ALL | reg_date,stat,up_reg_index | NULL | NULL | NULL | 2983126 | Using where; Using temporary; Using filesort |
+----+-------------+--------------+------+----------------------------+------+---------+------+---------+----------------------------------------------+
And the keys that can be used:
+--------------+------------+--------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------------+------------+--------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| stats_refers | 0 | PRIMARY | 1 | id | A | 2983126 | NULL | NULL | | BTREE | | |
| stats_refers | 0 | reg_date | 1 | reg_date | A | 13317 | NULL | NULL | | BTREE | | |
| stats_refers | 0 | reg_date | 2 | up_time | A | 1491563 | NULL | NULL | | BTREE | | |
| stats_refers | 0 | reg_date | 3 | refer | A | 2983126 | NULL | NULL | | BTREE | | |
| stats_refers | 1 | stat | 1 | reg_date | A | 15142 | NULL | NULL | | BTREE | | |
| stats_refers | 1 | stat | 2 | refer | A | 28683 | NULL | NULL | | BTREE | | |
| stats_refers | 1 | refer_uptime | 1 | refer | A | 2307 | NULL | NULL | | BTREE | | |
| stats_refers | 1 | refer_uptime | 2 | up_time | A | 1491563 | NULL | NULL | | BTREE | | |
| stats_refers | 1 | up_reg_index | 1 | reg_date | A | 2314 | NULL | NULL | | BTREE | | |
| stats_refers | 1 | up_reg_index | 2 | up_time | A | 1491563 | NULL | NULL | | BTREE | | |
+--------------+------------+--------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
And here is the table description:
CREATE TABLE `stats_refers` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`reg_date` int(10) unsigned NOT NULL,
`up_time` int(10) unsigned NOT NULL,
`refer` varchar(16) NOT NULL DEFAULT '',
`visits_count` int(10) unsigned NOT NULL DEFAULT '0',
`register_count` int(10) unsigned NOT NULL DEFAULT '0',
`players_count` int(10) unsigned NOT NULL DEFAULT '0',
`activity_count` int(10) unsigned NOT NULL DEFAULT '0',
`payment_users_count` int(10) unsigned NOT NULL DEFAULT '0',
`payment_count` int(10) unsigned NOT NULL DEFAULT '0',
`payment_sum` int(10) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
UNIQUE KEY `reg_date` (`reg_date`,`up_time`,`refer`),
KEY `stat` (`reg_date`,`refer`),
KEY `refer_uptime` (`refer`,`up_time`),
KEY `up_reg_index` (`reg_date`,`up_time`)
) ENGINE=InnoDB AUTO_INCREMENT=4136504 DEFAULT CHARSET=utf8 |
+----+-------------+--------------+-------+-----------------------------------+-------------+---------+------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------+-------+-----------------------------------+-------------+---------+------+---------+-------------+
| 1 | SIMPLE | stats_refers | index | reg_date,up_time,reg,search_index | group_index | 54 | NULL | 3011896 | Using where |
+----+-------------+--------------+-------+-----------------------------------+-------------+---------+------+---------+-------------+

Answer 1

The compound index stat, which I assume was created for use for this grouping operation, is in the reverse order for doing the grouping operation that you are trying to perform. To have an index available for this grouping operation, you would need to do one of the following:
reverse the column order on the stat index
add an individual index on refer
add another compound index on refer, reg_date
What decision you make would ultimately need to consider other query operations on the table.
You may want to think more broadly about your index usage here. While using compound indexes can improve performance vs. using individual indexes for each field that might need to be used across the range of queries on the table, in your case, you are replicating indexes on the same fields in different combinations within your table description. Without understanding all query use cases you might have it would be hard to look at overall indexing recommendations, but I just wanted to point out that you may want to think about this.
For example, there would be no apparent need at all for your up_reg_index as that indexing is already covered by the reg_date unique index. You might be best served with a set of indexes like this:
PRIMARY KEY (`id`),
UNIQUE KEY `regdate_uptime_refer` (`reg_date`,`up_time`,`refer`),
KEY `reg_date` (`reg_date`),
KEY `refer` (`refer`),
KEY `up_time` (`up_time`),
This would certainly require less space for your indexes than what you currently have and would allow for more flexibility in how to filter/join/group on these columns, but don't take this as a firm recommendation. Test the performance of different indexing scenarios against your different query scenarios (particularly insert performance if that is a use case).

Answer 2

INDEX(refer, reg_date)
in that order might get the optimizer to use that for the GROUP BY.
INDEX(up_time)
might be useful for the part of the WHERE that references up_time.
reg_date < 1435006800
AND up_time < 1435006800
The optimizer will think about using INDEX(reg_date, ...) or INDEX(up_time, ...), but only if the 'range' is less than, say, 20% of the table. It is very unlikely to try to use two separate indexes and do a "index merge".
Because of the "ranges" in the WHERE, it is not possible to have an index that handles both the WHERE and the ORDER BY.
No other indexes are useful for that query.
This 20% is imprecise. It exists because if too much of the table needs to be looked at, it is actually faster to do a table scan than to use the index.
You are selecting up_time, but not grouping by it. That's naughty. The query can deliver whatever up_time it feels like.
Why have an id, since you have a UNIQUE key that could be the PRIMARY KEY? I ask because the "clustering" of the PK could lead to this being the best option:
PRIMARY KEY(refer, reg_date, uptime)
(Of course, changing just that might mess up other queries.)