MySQL query slow (DISTINCT WHERE on indexed column) - mysql

Question

Specs:
MySQL version: 5.6.19 (Ubuntu)
Also tried MariaDB, and got the same problem
Table:
CREATE TABLE `x` (
`id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`a` INT(10) UNSIGNED NOT NULL,
`time` DECIMAL(16,6) NOT NULL,
PRIMARY KEY (`id`),
INDEX `a` (`a`),
INDEX `time` (`time`),
INDEX `time_a` (`time`, `a`)
)
COLLATE='utf8_unicode_ci'
ENGINE=InnoDB
AUTO_INCREMENT=298846
;
Query:
SELECT COUNT(DISTINCT `a`) c
FROM `x`
WHERE `time` >= (UNIX_TIMESTAMP()- (60 * 24));
This query is very SLOW, if there are a lot of rows with time in the given range. Also note that while there might be a lot of matching rows (thousands or tens of thousands or more), the amount of DISTINCT a's is always rather small (a few hundred).
The query is fast (basically instant), no matter the size of the table, when:
there are only few rows with time in the given range or when
there is no WHERE part (because of the index on a)
That makes me think that it is somehow unable to use the index on a when counting, even though EXPLAIN mentions all three indices in possibly_keys.
The issue remains even if:
time is of type BIGINT or DATETIME (with corresponding changes to the query)
ENGINE=MyISAM
Any suggestions?

Answer 1

SELECT COUNT(DISTINCT `a`)
FROM `x`;
will leapfrog through INDEX(a). See EXPLAIN FORMAT=JSON SELECT ... and look for "using_index_for_group_by": true. This makes it quite fast when there are only a small number of distinct a values.
I suspect that using the WHERE clause will say "using_index_for_group_by": "scanning", implying that it is less efficient. I suspect the implementers did the single-key case, but not the multi-key case.
Was that the entire table definition? I see AUTO_INCREMENT without any index for it. What's up? About the only difference between MyISAM and InnoDB that is relevant to this discussion is the handling of the PRIMARY KEY.
The datatype of time is probably not significant.
If I have not satisfied your "Any suggestions?" question, please rephrase the question.

Answer 2

Try using index hinting to force the query to use the index you want it to use.
SELECT COUNT(DISTINCT `a`) c
FROM `x` FORCE INDEX (the_index_you_want_to_use)
WHERE `time` >= (UNIX_TIMESTAMP()- (60 * 24));

Answer 3

It is best not to do any computations in such where clauses.
var unixtime = UNIX_TIMESTAMP()- (60 * 24)
SELECT COUNT(DISTINCT `a`) c
FROM `x` FORCE INDEX (the_index_you_want_to_use)
WHERE `time` >= unixtime

Answer 4

If I had to guess, the problem is types. UNIX_TIMESTAMP() returns an unsigned integer. Your time variable is decimal. These are not the same thing. And, type mismatches can confuse the optimizer.
It sounds like the table is big, so changing the type is not feasible (you might want to test this, though, if you can by selecting the data into a new table with the right types).
The following might help:
WHERE `time` >= cast(UNIX_TIMESTAMP() - (60 * 24) as unsigned);
You could also declare a local unsigned variable and store the "constant" in the variable to see if that fixes the performance problem.
Finally, if the index on time, a is not being used, try this variation of the query:
SELECT COUNT(*) as c
FROM (SELECT DISTINCT a
FROM `x`
WHERE `time` >= CAST(unixtime - 24 * 60 as unsigned)
) ax
I have seen this re-structuring improve performance on other databases, although not on MySQL.