Incorrect ordering on query with group by clause - mysql

Question

So I have the following query:
SELECT sensor.id as `sensor_id`,
sensor_reading.id as `reading_id`,
sensor_reading.reading as `reading`,
from_unixtime(sensor_reading.reading_timestamp) as `reading_timestamp`,
sensor_reading.lower_threshold as `lower_threshold`,
sensor_reading.upper_threshold as `upper_threshold`,
sensor_type.units as `unit`
FROM sensor
LEFT JOIN sensor_reading ON sensor_reading.sensor_id = sensor.id
LEFT JOIN sensor_type ON sensor.sensor_type_id = sensor_type.id
WHERE sensor.company_id = 1
GROUP BY sensor_reading.sensor_id
ORDER BY sensor_reading.reading_timestamp DESC
There are three tables in play here. A sensor_type table, which is just used for a single display field (units), a sensor table, which contains information on a sensor, and a sensor_reading table, which contains the individual readings for a sensor. There are multiple readings which apply to a single sensor, and so each entry in the sensor_reading table has a sensor_id which is linked to the ID field in the sensor table with a foreign key constraint.
In theory, this query should return the most recent sensor_reading for EACH unique sensor. Instead, it's returning the first reading for each sensor instead. I've seen a few posts on here with similar issues, but haven't been able to resolve this using any of their answers. Ideally, the query needs to be as efficient as possible, as this table has several thousand readings (and continues to grow).
Does anyone know how I might change this query to return the most recent reading? If I remove the GROUP BY clause, it returns the right order, but I then have to sift through the data to get the most recent for each sensor.
Ideally, I don't want to run sub-queries as this slows things down a lot, and speed is a big factor here.
Thanks!

Answer 1

In theory, this query should return the most recent sensor_reading for EACH unique sensor.
This is a fairly common misconception with the MySQL Group by extension, that allows you to select columns with no aggregation that are not contained in the group by clause. What the documentation states is:
The server is free to choose any value from each group, so unless they are the same, the values chosen are indeterminate. Furthermore, the selection of values from each group cannot be influenced by adding an ORDER BY clause
So since you are grouping by sensor_reading.sensor_id, MySQL will chose any row from sensor_reading for each sensor_id, then after choosing one row for each sensor_id it will then apply the ordering to the rows that are chosen.
Since you only want the latest row for each sensor, the general approach would be:
SELECT *
FROM sensor_reading AS sr
WHERE NOT EXISTS
( SELECT 1
FROM sensor_reading AS sr2
WHERE sr2.sensor_id = sr.sensor_id
AND sr2.reading_timestamp > sr.reading_timestamp
);
However, MySQL will optimise LEFT JOIN/IS NULL better than NOT EXISTS so a MySQL specific solution would be:
SELECT sr.*
FROM sensor_reading AS sr
LEFT JOIN sensor_reading AS sr2
ON sr2.sensor_id = sr.sensor_id
AND sr2.reading_timestamp > sr.reading_timestamp
WHERE sr2.id IS NULL;
So incorporating this into your query, you would end up with:
SELECT sensor.id as `sensor_id`,
sensor_reading.id as `reading_id`,
sensor_reading.reading as `reading`,
from_unixtime(sensor_reading.reading_timestamp) as `reading_timestamp`,
sensor_reading.lower_threshold as `lower_threshold`,
sensor_reading.upper_threshold as `upper_threshold`,
sensor_type.units as `unit`
FROM sensor
LEFT JOIN sensor_reading
ON sensor_reading.sensor_id = sensor.id
LEFT JOIN sensor_type
ON sensor.sensor_type_id = sensor_type.id
LEFT JOIN sensor_reading AS sr2
ON sr2.sensor_id = sensor_reading.sensor_id
AND sr2.reading_timestamp > sensor_reading.reading_timestamp
WHERE sensor.company_id = 1
AND sr2.id IS NULL
ORDER BY sensor_reading.reading_timestamp DESC;
An alternative method for getting the maximum per group is to inner join back to the latest row, so something like:
SELECT sr.*
FROM sensor_reading AS sr
INNER JOIN
( SELECT sensor_id, MAX(reading_timestamp) AS reading_timestamp
FROM sensor_reading
GROUP BY sensor_id
) AS sr2
ON sr2.sensor_id = sr.sensor_id
AND sr2.reading_timestamp = sr.reading_timestamp;
You may find that this is more efficient than the other method, or you may not, YMMV. It basically depends on your data and indexes, and as you have said, subqueries can be an issue in MySQL due to the fact that the full result is matierialised initially.