Limit on join query using between - mysql

Question

I am trying to filter some results indexed by a timestamp using another set of results that define valid timestamp periods.
Current query:
SELECT Measurements.moment AS "moment",
Measurements.actualValue,
start,
stop
FROM Measurements
INNER JOIN (SELECT COALESCE(#previousValue <> M.actualValue AND #previousResource = M.resourceId, 1) AS "changed",
(COALESCE(#previousMoment, ?)) AS "start",
M.moment AS "stop",
#previousValue AS "actualValue",
M.resourceId,
#previousMoment := moment,
#previousValue := M.actualValue,
#previousResource := M.resourceId
FROM Measurements `M`
INNER JOIN (SELECT #previousValue := NULL, #previousResource := NULL, #previousMoment := NULL) `d`
WHERE (M.moment BETWEEN ? AND ?) AND
(M.actualValue > ?)
ORDER BY M.resourceId ASC, M.moment ASC) `changes` ON Measurements.moment BETWEEN changes.start AND changes.stop
WHERE (Measurements.resourceId = 1) AND
(Measurements.moment BETWEEN ? AND ?) AND
(changes.changed)
ORDER BY Measurements.moment ASC;
resourceId, moment is already an index.
Since these are actually timeseries data, is there any way to limit the join on just 1 match to improve performance?
Sample data
+-------------+---------------------+------------+
| actualValue | moment | resourceId |
+-------------+---------------------+------------+
| 0.01 | 2018-09-26 07:50:25 | 1 |
| 0.01 | 2018-09-26 07:52:35 | 1 |
| 0.01 | 2018-09-26 07:52:44 | 2 |
| 0.01 | 2018-09-26 07:52:54 | 1 |
| 0.01 | 2018-09-26 07:53:03 | 1 |
| 0.01 | 2018-09-26 07:53:13 | 2 |
| 0.01 | 2018-09-26 07:53:22 | 1 |
| 0.01 | 2018-09-26 07:54:32 | 1 |
| 0.01 | 2018-09-26 07:55:41 | 1 |
| 0.01 | 2018-09-26 07:56:51 | 1 |
+-------------+---------------------+------------+
Expected output: All measurements with resourceId=1 where resourceId=2 had a measurement in that same minute (in an advanced version, the minute can be dynamic).
+-------------+---------------------+------------+
| actualValue | moment | resourceId |
+-------------+---------------------+------------+
| 0.01 | 2018-09-26 07:52:35 | 1 |
| 0.01 | 2018-09-26 07:52:54 | 1 |
| 0.01 | 2018-09-26 07:53:03 | 1 |
| 0.01 | 2018-09-26 07:53:22 | 1 |
+-------------+---------------------+------------+

Answer 1

When you use an independent subquery (this case) then it's executed entirely before the outer query. In your case this could be potentially massive, and probably most of the rows are not really neeeded.
If you rephrase the query using an inner JOIN then the secondary access to the table will be filtered out immediately, avoiding the need for a full scan of the table.
Try the following query:
select
m.moment,
m.actualValue,
c.moment as start,
timestampadd(minute, 1, c.moment) as stop
from Measurements m
join Measurements c on m.moment
between c.moment and timestampadd(minute, 1, c.moment)
where m.resourceId = 1
and c.resourceId = 2
and m.moment between ? and ?
order by m.moment

Answer 2

Composite Index needed:
Measurements: INDEX(resourceId, moment) -- in this order
You may want AND (Measurements.moment BETWEEN ? AND ?) in the subquery
In a "derived table" (the subquery you have), the Optimizer is free to ignore the ORDER BY. If, however, you add a LIMIT, the ORDER BY will be honored.

Answer 3

I found a solution using table unpivoting:
SELECT moment, value
FROM (SELECT IF(resourceId = ? AND #previousValue = 0, NULL, actualValue) AS value,
measurements.moment,
resourceId,
#previousValue := IF(resourceId <> ?, actualValue, #previousValue) AS enabled
FROM (SELECT *
FROM (SELECT moment,
Measurements.actualValue,
Measurements.resourceId AS resourceId
FROM Measurements
WHERE Measurements.resourceId = ?
AND moment BETWEEN ? AND ?
UNION (SELECT start,
periods.actualValue AS actualValue,
resourceId
FROM (SELECT COALESCE(#previousValue <> M3.actualValue, 1) AS "changed",
(COALESCE(#previousMoment, ?)) AS "start",
#previousMoment := M3.moment AS "stop",
COALESCE(#previousValue, IF(M3.actualValue = 1, 0, 1)) AS "actualValue",
M3.resourceId AS resourceId,
#previousValue := M3.actualValue
FROM Measurements `M3`
INNER JOIN (SELECT #previousValue := NULL,
#previousMoment := NULL) `d`
WHERE (M3.moment BETWEEN ? AND ?)
ORDER BY M3.resourceId ASC, M3.moment ASC) AS periods
WHERE periods.changed)) AS measurements
ORDER BY moment ASC) AS measurements
INNER JOIN (SELECT #previousValue := NULL) `k`) AS mixed
WHERE value IS NOT NULL
AND resourceId = ?;
This runs essentially runs table once per select, running ~40k x ~4k rows in 100ms.