Optimisation in badly-performing query - mysql

Question

I have the following query which is working correctly, however it is running very poorly. I am suspecting that my issue is with the two comparison conditions in the INNER JOIN statement. Both of the fields have an index, but the query optimiser in MySQL seems to be ignoring them. Here is my query:
EDIT: Changed query to use the one suggested below by Gordon, as it has kept the same results but is performing faster. EXPLAIN statement is still not happy though, and the output is shown below.
SELECT a.id
FROM pc a INNER JOIN
(SELECT correction_value, MAX(seenDate) mxdate
FROM pc FORCE INDEX (IDX_SEENDATE)
WHERE seenDate BETWEEN '2017-03-01' AND '2017-04-01'
GROUP BY correction_value
) b
ON a.correction_value = b.correction_value AND
a.seenDate = b.mxdate INNER JOIN
cameras c
ON c.camera_id = a.camerauid
WHERE c.in_out = 0;
EXPLAIN
+----+-------------+------------+------------+-------+-------------------+--------------+---------+----------+---------+----------+---------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+-------------------+--------------+---------+----------+---------+----------+---------------------------------------+
| 1 | PRIMARY | <derived2> | NULL | ALL | NULL | NULL | NULL | NULL | 2414394 | 100 | Using where; |
| | | | | | | | | | | | Using temporary; |
| | | | | | | | | | | | Using filesort |
+----+-------------+------------+------------+-------+-------------------+--------------+---------+----------+---------+----------+---------------------------------------+
| 1 | PRIMARY | a | NULL | ref | correction_value, | idx_seenDate | 5 | b.mxdate | 1 | 3.8 | Using where |
| | | | | | idx_seenDate, | | | | | | |
| | | | | | fk_camera_idx | | | | | | |
+----+-------------+------------+------------+-------+-------------------+--------------+---------+----------+---------+----------+---------------------------------------+
| 1 | PRIMARY | c | NULL | ALL | PRIMARY | NULL | NULL | NULL | 41 | 2.44 | Using where; |
| | | | | | | | | | | | Using join buffer (Block Nested Loop) |
+----+-------------+------------+------------+-------+-------------------+--------------+---------+----------+---------+----------+---------------------------------------+
| 2 | DERIVED | pc | NULL | range | correction_value, | idx_seenDate | 5 | NULL | 2414394 | 100 | Using index Condition; |
| | | | | | idx_seenDate | | | | | | Using temporary; |
| | | | | | | | | | | | Using filesort |
+----+-------------+------------+------------+-------+-------------------+--------------+---------+----------+---------+----------+---------------------------------------+
How can the query be optimised but still have the same outcome?

Answer 1

Let's start by focusing on the subquery.
SELECT correction_value,
MAX(seenDate) mxdate
FROM pc
WHERE seenDate BETWEEN '2017-03-01' AND '2017-04-01'
GROUP BY correction_value
Please run that with twice, with
INDEX sc (seenDate, correction_value)
INDEX cs (correction_value, seenDate)
Please FORCE one index, then the other. Depending on what version of MySQL you are running, one of the indexes will work better than the other.
I think that later versions will prefer "cs" because it can leapfrog through the index very efficiently.
Once you have determined which composite index to use, then remove the FORCE and the unused index, then try the entire query. The same index should do fine for the combined query.
Since your task seems to involve a "groupwise max", I suggest you see if there are performance tips here: http://mysql.rjweb.org/doc.php/groupwise_max

Answer 2

Try this
SELECT
a.id
FROM pc a
INNER JOIN
(SELECT correction_value, MAX(seenDate) mxdate
FROM pc
INNER JOIN cameras ON (cameras.camera_id = pc.camerauid AND cameras.in_out = 0)
WHERE pc.seenDate BETWEEN '2017-03-01' AND '2017-04-01'
GROUP BY correction_value) b ON (a.correction_value = b.correction_value AND a.seenDate = b.mxdate);
use index on pc.seenDate column.

Answer 3

I would start by writing the query as:
SELECT a.id
FROM pc a INNER JOIN
(SELECT correction_value, MAX(seenDate) mxdate
FROM pc
WHERE seenDate BETWEEN '2017-03-01' AND '2017-04-01'
GROUP BY correction_value
) b
ON a.correction_value = b.correction_value AND
a.seenDate = b.mxdate INNER JOIN
cameras c
ON c.camera_id = a.camerauid
WHERE c.in_out = 0; - don't use single quotes if `in_out` is a number
The place to start with this query is to have indexes: pc(seendate, correction_value, seendate) and cameras(camera_id, in_out).
There may also be ways to rewrite the query, if that is not sufficient.

Answer 4

RDBMS uses the output of the first query as a input of the next query. So, if we look at the derived query, it is using a filter so we can use that as a first query, then join to the pc then join to the camera table.
Indexes: mentioned by Gordon Linof or pc(id, correction_value, seendate) and cameras(camera_id, in_out)
The final query could be rewrite as below:
SELECT a.id
--add any other column here, you want to show in the EXPLAINED output
FROM
(
SELECT id, correction_value, MAX(seenDate) mxdate
FROM pc
WHERE seenDate BETWEEN '2017-03-01' AND '2017-04-01'
GROUP BY correction_value
) a
INNER JOIN pc b
ON a.correction_value = b.correction_value
AND a.seenDate = b.mxdate
INNER JOIN cameras c
ON c.camera_id = a.camerauid
WHERE c.in_out = 0;

Answer 5

From your question it is not clear how the tables are indexed, but in this subquery
(SELECT correction_value, MAX(seenDate) mxdate
FROM pc FORCE INDEX (IDX_SEENDATE)
WHERE seenDate BETWEEN '2017-03-01' AND '2017-04-01'
GROUP BY correction_value
) b
you want to have a composite index on both fields seenDate, correction_value:
CREATE INDEX seenCorr_ndx ON pc (seenDate, correction_value);
(you can drop any index on seenDate alone, and I expect you do not need the FORCE INDEX either).
You may end up needing two composite indexes, one with seenDate first, one with correction_value first.