Aligning timestamps when not quite synchronized - mysql

Question

I have 3 processes A, B and C as defined in the following series of tables:
http://sqlfiddle.com/#!2/48f54
CREATE TABLE processA
(date_time datetime, valueA int);
INSERT INTO processA
(date_time, valueA)
VALUES
('2013-1-8 22:10:00', 100),
('2013-1-8 22:15:00', 100),
('2013-1-8 22:30:00', 100),
('2013-1-8 22:35:00', 100),
('2013-1-8 22:40:00', 100),
('2013-1-8 22:45:00', 100),
('2013-1-8 22:50:00', 100),
('2013-1-8 23:05:00', 100),
('2013-1-8 23:10:00', 100),
('2013-1-8 23:20:00', 100),
('2013-1-8 23:25:00', 100),
('2013-1-8 23:35:00', 100),
('2013-1-8 23:40:00', 100),
('2013-1-9 00:05:00', 100),
('2013-1-9 00:10:00', 100);
CREATE TABLE processB
(date_time datetime, valueB decimal(4,2));
INSERT INTO processB
(date_time, valueB)
VALUES
('2013-1-08 21:46:00', 3),
('2013-1-08 22:11:00', 4),
('2013-1-08 22:31:00', 5),
('2013-1-08 22:36:00', 6),
('2013-1-08 22:41:00', 7),
('2013-1-08 23:06:00', 8),
('2013-1-08 23:20:00', 2),
('2013-1-08 23:46:00', 3),
('2013-1-09 00:34:00', 9);
CREATE TABLE processC
(date_time datetime, status varchar(4));
INSERT INTO processC
VALUES
('2013-1-08 18:00:00', 'yes'),
('2013-1-08 19:00:00', 'yes'),
('2013-1-08 20:00:00', 'yes'),
('2013-1-08 21:00:00', 'yes'),
('2013-1-08 22:00:00', 'yes'),
('2013-1-08 23:00:00', 'no'),
('2013-1-08 00:00:00', 'no'),
('2013-1-08 01:00:00', 'no');
As you can see the time at which readings occur for each of the processes is not the same.
ProcessA, IF it occurs, does so at 5 minute intervals
ProcessB, readings occur at unpredictable times but usually occur multiple times within the hour
ProcessC will always have an hourly value (yes or no).
Firstly, I want to convert processB so that there is a reading at ever 5 minute interval so the data aligns with processA, which can then enable me to do a simple join of both tables at the 5 minute interval mark. For the conversion, the data at every 5 minutes should be set to the nearest processB observation available within [-30,30) minute window. If values are equidistant then take the average. If none is available in the 30 minute window then set it to null.
Once I have that, I can do a simple join on %Y%m%d%H with ProcessC using something like the following to get a final table with all data aligned at the 5 minute interval mark:
date_format(date_time, '%Y%m%d%H') = date_format(date_time, '%Y%m%d%H')
If anyone has any pointers/guidance I would appreciate some direction. I appreciate it.
Sample output:
'2013-1-8 22:10:00', 100, 4, yes <--- closer to 22:11 than 21:46
'2013-1-8 22:15:00', 100, 4, yes <--- closer to 22:11 than 21:31
'2013-1-8 22:30:00', 100, 5, yes <--- closer to 22:31 than 22:11
'2013-1-8 22:35:00', 100, 6, yes <--- closer to 22:36 than 22:31
'2013-1-8 22:40:00', 100, 7, yes <--- closer to 22:41 than 22:36
'2013-1-8 22:45:00', 100, 7, yes <--- closer to 22:41 than 23:06
'2013-1-8 22:50:00', 100, 7, yes <--- closer to 22:41 than 23:06
'2013-1-8 23:05:00', 100, 8, yes <--- closer to 23:06 than 23:06
'2013-1-8 23:10:00', 100, 8, no <--- closer to 23:06 than 23:20
'2013-1-8 23:20:00', 100, 2, no <--- closer to 23:20 than 23:10
'2013-1-8 23:25:00', 100, 2, no <--- closer to 23:20 than 23:10
'2013-1-8 23:35:00', 100, 3, no <--- closer to 23:46 than 23:20
'2013-1-9 00:05:00', 100, 3, no <--- closer to 23:46 than 00:34
'2013-1-9 00:10:00', 100, 6, no <--- takes the avg of 3 and 9

Answer 1

The tricky part of this is the retrieval of the appropriate row or rows from processB that correspond to each row of processA as you figured out.
Let's take it step by step.
First, we need to be able to join processA and processB to retrieve the candidate timestamp pairs. Let's do it like this:
SELECT a.date_time a,
TIMESTAMPDIFF(SECOND, a.date_time, b.date_time) timediff
FROM processA a
JOIN processB b
ON TIMESTAMPDIFF(SECOND, a.date_time, b.date_time) >= -1800
AND TIMESTAMPDIFF(SECOND, a.date_time, b.date_time) < 1800
This gets us the a and b times meeting the [-30, 30) criterion. There are a lot of rows in this result; but we can inspect it to make sure we've done the range comparison correctly. http://sqlfiddle.com/#!2/48f54/47/0
Now we need to generate the time window to search for each a record for your one or more matching b records. Like so.
SELECT a,
MIN(ABS(timediff)) windowsize
FROM (
SELECT a.date_time a,
TIMESTAMPDIFF(SECOND, a.date_time, b.date_time) timediff
FROM processA a
JOIN processB b
ON TIMESTAMPDIFF(SECOND, a.date_time, b.date_time) >= -1800
AND TIMESTAMPDIFF(SECOND, a.date_time, b.date_time) < 1800
) d
GROUP BY a
This yields two columns: the first is the timestamp from a, and the second is the time range of the nearest b timestamp (or timestamps, if more than one are to be averaged) that are in range. This resultset doesn't have any row for a records that don't have b records near enough to consider. http://sqlfiddle.com/#!2/48f54/46/0
Finally, we need to retrieve and average the b record values for each a record. Here this is.
SELECT processA.date_time date_time,
processA.valueA valueA,
AVG(processB.valueB) valueB
FROM processA
LEFT JOIN (
SELECT a,
MIN(ABS(timediff)) windowsize
FROM (
SELECT a.date_time a,
TIMESTAMPDIFF(SECOND, a.date_time, b.date_time) timediff
FROM processA a
JOIN processB b
ON TIMESTAMPDIFF(SECOND, a.date_time, b.date_time) >= -1800
AND TIMESTAMPDIFF(SECOND, a.date_time, b.date_time) < 1800
) d
GROUP BY a
) j ON processA.date_time = j.a
LEFT JOIN processB ON ( processB.date_time >= j.a - INTERVAL j.windowsize SECOND
AND processB.date_time <= j.a + INTERVAL j.windowsize SECOND
AND processB.date_time < j.a + INTERVAL 1800 SECOND)
GROUP BY processA.date_time, processA.valueA
Notice there are a couple of open ranges here (< operators instead of <= operators). Those are there to accomodate your [-30, 30) open range. Here's the query. http://sqlfiddle.com/#!2/48f54/45/0
This final query joins together three tables: processA, our virtual table showing the search range for each timestamp, and process B. The last ON clause performs the actual range search. It's made slightly more complicated by the open range.
See how this goes? It's helpful to construct the query from the inside out.
Don't forget to put an index on processB.date_time.
I am taking the liberty of leaving the join of processC to this virtual table to you.