SQL MAX and GROUP BY with WHERE - mysql

Question

Given the following table:
CREATE TABLE `test` (
`id` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
`device_id` INT(11) UNSIGNED NOT NULL,
`distincted` BIT(1) NOT NULL DEFAULT b'0',
`timestamp_detected` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `idx1` (`device_id`),
INDEX `idx2` (`device_id`, `timestamp_detected`),
CONSTRAINT `test_ibfk_1` FOREIGN KEY (`device_id`) REFERENCES `device` (`id`)
)
COLLATE='utf8mb4_general_ci'
ENGINE=InnoDB
ROW_FORMAT=COMPACT;
I want to perform a groupwise max on timestamp_detected grouped by device_id with the following:
SELECT lh1.id, lh1.timestamp_detected, lh1.device_id FROM test as lh1,
(SELECT MAX(timestamp_detected) as max_timestamp_detected, device_id FROM test GROUP BY device_id) as lh2
WHERE lh1.timestamp_detected = lh2.max_timestamp_detected
AND lh1.device_id = lh2.device_id;
This yields the following results when run with explain:
+----+-------------+------------+-------+---------------------------------------------------------+------------------------------+---------+------------------------------------------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+-------+---------------------------------------------------------+------------------------------+---------+------------------------------------------+------+--------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 15 | Using where |
| 1 | PRIMARY | lh1 | ref | FK_location_history_device,device_id_timestamp_detected | device_id_timestamp_detected | 9 | lh2.device_id,lh2.max_timestamp_detected | 1 | Using index |
| 2 | DERIVED | test | range | FK_location_history_device,device_id_timestamp_detected | device_id_timestamp_detected | 4 | NULL | 15 | Using index for group-by |
+----+-------------+------------+-------+---------------------------------------------------------+------------------------------+---------+------------------------------------------+------+--------------------------+
Now there is a requirement that only those rows with distincted = 1 should be included in the results. I modified the query to the following:
SELECT lh1.id, lh1.timestamp_detected, lh1.device_id FROM test as lh1,
(SELECT MAX(timestamp_detected) as max_timestamp_detected, device_id FROM test WHERE distincted = 1 GROUP BY device_id) as lh2
WHERE lh1.timestamp_detected = lh2.max_timestamp_detected
AND lh1.device_id = lh2.device_id;
It returns the results correctly however it seems to take longer. Running an explain yields the following:
+----+-------------+------------+-------+---------------------------------------------------------+------------------------------+---------+------------------------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+-------+---------------------------------------------------------+------------------------------+---------+------------------------------------------+------+-------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 860 | Using where |
| 1 | PRIMARY | lh1 | ref | FK_location_history_device,device_id_timestamp_detected | device_id_timestamp_detected | 9 | lh2.device_id,lh2.max_timestamp_detected | 1 | Using index |
| 2 | DERIVED | test | index | FK_location_history_device,device_id_timestamp_detected | FK_location_history_device | 4 | NULL | 860 | Using where |
+----+-------------+------------+-------+---------------------------------------------------------+------------------------------+---------+------------------------------------------+------+-------------+
I tried adding the distincted column to index idx2 to no avail. How can I optimize this query?

Answer 1

The query is:
SELECT lh1.id, lh1.timestamp_detected, lh1.device_id
FROM test lh1 JOIN
(SELECT MAX(timestamp_detected) as max_timestamp_detected, device_id
FROM test
WHERE distincted = 1
GROUP BY device_id
) as lh2
on lh1.timestamp_detected = lh2.max_timestamp_detected AND
lh1.device_id = lh2.device_id;
For this query, I would suggest indexes on test(distincted, device_id, time_stamp_detected) and test(device_id, timestamp_detected).
I also wonder if you would get better performance with this equivalent query:
SELECT lh1.id, lh1.timestamp_detected, lh1.device_id
FROM test lh1
WHERE distincted = 1 AND
NOT EXISTS (SELECT 1
FROM test t
WHERE t.distincted = 1 AND
t.device_id = lh1.device_id AND
t.timestamp_detected > lh1.timestamp_detected
);
And these two indexes: test(distincted) and test(device_id, timestamp_detected, distincted).