Why does MySQL not use the index from EXPLAIN? - mysql

Question

I have a straight forward table which currently has ~10M rows.
Here is the definition:
CREATE TABLE `train_run_messages` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`train_id` int(10) unsigned NOT NULL,
`customer_id` int(10) unsigned NOT NULL,
`station_id` int(10) unsigned NOT NULL,
`train_run_id` int(10) unsigned NOT NULL,
`timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`type` tinyint(4) NOT NULL,
`customer_station_track_id` int(10) unsigned DEFAULT NULL,
`lateness_type` tinyint(3) unsigned NOT NULL,
`lateness_amount` mediumint(9) NOT NULL,
`lateness_code` tinyint(3) unsigned DEFAULT '0',
`info_text` varchar(32) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `timestamp` (`timestamp`),
KEY `lateness_amount` (`lateness_amount`),
KEY `customer_timestamp` (`customer_id`,`timestamp`),
KEY `trm_customer` (`customer_id`),
KEY `trm_train` (`train_id`),
KEY `trm_station` (`station_id`),
KEY `trm_trainrun` (`train_run_id`),
KEY `FI_trm_customer_station_tracks` (`customer_station_track_id`),
CONSTRAINT `FK_trm_customer_station_tracks` FOREIGN KEY (`customer_station_track_id`) REFERENCES `customer_station_tracks` (`id`),
CONSTRAINT `trm_customer` FOREIGN KEY (`customer_id`) REFERENCES `customers` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `trm_station` FOREIGN KEY (`station_id`) REFERENCES `stations` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `trm_train` FOREIGN KEY (`train_id`) REFERENCES `trains` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `trm_trainrun` FOREIGN KEY (`train_run_id`) REFERENCES `train_runs` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=9928724 DEFAULT CHARSET=utf8;
We have lots of queries that filter by customer_id and timestamp so we have created a combined index for that.
Now I have this simple query:
SELECT * FROM `train_run_messages` WHERE `customer_id` = '5' AND `timestamp` >= '2013-12-01 00:00:57' AND `timestamp` <= '2013-12-31 23:59:59' LIMIT 0, 100
On our current machine with ~10M entries this query takes ~16 seconds, which is way to long in my taste, since there is an index for queries like this.
So lets look at the output of explain for this query:
+----+-------------+--------------------+------+------------------------------------------- +--------------------+---------+-------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------------+------+-------------------------------------------+--------------------+---------+-------+--------+-------------+
| 1 | SIMPLE | train_run_messages | ref | timestamp,customer_timestmap,trm_customer | customer_timestamp | 4 | const | 551405 | Using where |
+----+-------------+--------------------+------+-------------------------------------------+--------------------+---------+-------+--------+-------------+
So MySQL is telling me that it would use the customer_timestamp index, fine! Why does the query still take ~16 seconds?
Since I don't always trust the MySQL query analyzer lets try it with a forced index:
SELECT * FROM `train_run_messages` USE INDEX (customer_timestamp) WHERE `customer_id` = '5' AND `timestamp` >= '2013-12-01 00:00:57' AND `timestamp` <= '2013-12-31 23:59:59' LIMIT 0, 100
Query Time: 0.079s!!
Me: puzzled!
So can anyone explain why MySQL is obviously not using the index that it says it would use from the EXPLAIN output? And is there any way to prove what index it really used when performing the real query?
Btw: Here is the output from the slow-log:
# Time: 131217 11:18:04
# User#Host: root[root] # localhost [127.0.0.1]
# Query_time: 16.252878 Lock_time: 0.000168 Rows_sent: 100 Rows_examined: 9830711
SET timestamp=1387275484;
SELECT * FROM `train_run_messages` WHERE `customer_id` = '5' AND `timestamp` >= '2013-12-01 00:00:57' AND `timestamp` <= '2013-12-31 23:59:59' LIMIT 0, 100;
Alltough it does not specifically say that it is not using any index the Rows_examined suggests that it does a full tablescan.
So is this fixable without using USE INDEX? We are using Propel as ORM and there is currently no way to use MySQL-specific "USE INDEX" without manually writing the query.
Edit:
Here is the output of EXPLAIN and USE INDEX:
+----+-------------+--------------------+-------+--------------------+--------------------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------------+-------+--------------------+--------------------+---------+------+--------+-------------+
| 1 | SIMPLE | train_run_messages | range | customer_timestmap | customer_timestmap | 8 | NULL | 191264 | Using where |
+----+-------------+--------------------+-------+--------------------+--------------------+---------+------+--------+-------------+

Answer 1

MySQL has three candidate indexes
(timestamp)
(customer_id, timestamp)
(customer_id)
and you are asking
`customer_id` = '5' AND `timestamp` BETWEEN ? AND ?
The optimizer has choose (customer_id, timestamp) from statistics.
InnoDB Engine's optimizer depends on statistics which uses sampling when table is opend. default sampling reads 8 pages on index file.
So, I suggest three things as follows
increase innodb_stats_sample_pages=64.
Default value of innodb_stats_sample_pages is 8 pages.
refer to http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_stats_sample_pages
remove redandant index. following index is just fine. currently there is only customer_id = 5 (you said)
(timestamp)
(customer_id)
run OPTIMIZE TABLE train_run_messages to re-organize table.
this reduces table and index size and sometimes this makes optimizer smarter

Answer 2

To me, the biggest thing it is failing on your wrapping the customer ID in quotes... such as = '5'. By doing this, it cant use the customer/timestamp index because the customer Id needs to be converted to a string to match your '5' vs just = 5 and you should be good to go.