MySQL optimizer issue with index on two columns and limit clause - mysql

Question

We have this table:
CREATE TABLE `test_table` (
`id` INT NOT NULL AUTO_INCREMENT,
`time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
`value` FLOAT NOT NULL,
`session` INT NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `session_time_idx` (`session` ASC,`time` ASC)
) ENGINE = InnoDB;
It is used to store different "measurement sessions" each resulting in potentially hundreds of thousands of rows. Different measurement sessions may have the same or overlapping timestamp ranges. We then need to randomly access single measurements with queries like this:
SELECT * FROM `test_table` WHERE `session` = 2 AND `time` < '2003-12-02' ORDER BY `time` DESC LIMIT 1;
We need to query for times which are spread uniformly on the measurement session. The "less than" operator is necessary because we don't know exactly when each measurement was taken, we just need to find the last measurement which was performed before a given date and time.
Depending on the time specified in the query, we have 2 possible resulting plans:
mysql> EXPLAIN SELECT * FROM `test_table` WHERE `session` = 2 AND `time` < '2003-12-02' ORDER BY `time` DESC LIMIT 1;
+----+-------------+------------+-------+------------------+------------------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+-------+------------------+------------------+---------+------+------+-------------+
| 1 | SIMPLE | test_table | range | session_time_idx | session_time_idx | 8 | NULL | 6050 | Using where |
+----+-------------+------------+-------+------------------+------------------+---------+------+------+-------------+
mysql> EXPLAIN SELECT * FROM `test_table` WHERE `session` = 2 AND `time` < '2005-01-02' ORDER BY `time` DESC LIMIT 1;
+----+-------------+------------+------+------------------+------------------+---------+-------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+------+------------------+------------------+---------+-------+--------+-------------+
| 1 | SIMPLE | test_table | ref | session_time_idx | session_time_idx | 4 | const | 127758 | Using where |
+----+-------------+------------+------+------------------+------------------+---------+-------+--------+-------------+
The first plan uses the whole index (session and time), typically resulting in sub-ms execution times on development machines.
The second plan uses only part of the index then scans the results of the whole session, sometimes hundreds of thousands of rows. Needless to say, the performance of the second plan is very poor. Tens of ms on development machines, which can become seconds on the slow production embedded devices.
The difference between the two queries is just the amount of rows which would match the query if no "LIMIT" clause was used. This makes sense when no "LIMIT" is specified because scanning the data directly can be an advantage instead of scanning both the second part of the index and the data. But MySQL doesn't seem to care about the fact that we only need one row: using the full index seems to be always the best choice in this case.
I made some tests which resulted in the following observations:
if I select just "id", "time" and/or "session" (not "value") the full index is used in all cases (because all needed data is in the indexes); so, while slightly cumbersome, querying first the "id" and then the rest of the data would work
using "FORCE INDEX (session_time_idx)" does fix the bad plan and results in fast queries all the times
no issue is present when using a single column index on time
running OPTIMIZE TABLE does not make any difference
using MyIASM instead if InnoDB makes no difference
using a simple integer instead of a TIMESTAMP makes no difference (as expected: TIMESTAMP is an integer after all)
I played with various parameters, including "max_seeks_for_key", but I couldn't fix the bad plan
Since we are using this kind of access pattern in many places and we have a custom ORM system, I'd like to know if there is a way to "convince" MySQL to do the right thing without having to add "FORCE INDEX" support to the ORM.
Any other suggestion for working around this problem would also be appreciated.
My setup: MySQL Server 5.5.47 on Ubuntu 14.04 64-bit.
Update: this also happens with MySQL Server 5.6 and 5.7.
For reference, this is the script I am using to create the test setup:
set ##time_zone = "+00:00";
drop schema if exists `index_test`;
create schema `index_test`;
use `index_test`;
CREATE TABLE `test_table` (
`id` INT NOT NULL AUTO_INCREMENT,
`time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
`value` FLOAT NOT NULL,
`session` INT NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `session_time_idx` (`session` ASC,`time` ASC)
) ENGINE = InnoDB;
delimiter $$
CREATE PROCEDURE fill(total int)
BEGIN
DECLARE count int;
DECLARE countPerSs int;
DECLARE tim int;
set count = 0;
set countPerSs = 100000;
set tim = unix_timestamp('2000-01-01');
myloop: LOOP
insert into `test_table` set `value` = rand(), `session` = count div countPerSs, `time` = from_unixtime(tim);
set tim = tim + 10 * 60;
SET count = count + 1;
IF count < total THEN
ITERATE myloop;
END IF;
LEAVE myloop;
END LOOP myloop;
END;
$$
delimiter ;
call fill(500000);