Interpolate missing values in a MySQL table - mysql

Question

I have some intraday stock data saved into a MySQL table which looks like this:
+----------+-------+
| tick | quote |
+----------+-------+
| 08:00:10 | 5778 |
| 08:00:11 | 5776 |
| 08:00:12 | 5778 |
| 08:00:13 | 5778 |
| 08:00:14 | NULL |
| 08:00:15 | NULL |
| 08:00:16 | 5779 |
| 08:00:17 | 5778 |
| 08:00:18 | 5780 |
| 08:00:19 | NULL |
| 08:00:20 | 5781 |
| 08:00:21 | 5779 |
| 08:00:22 | 5779 |
| 08:00:23 | 5779 |
| 08:00:24 | 5778 |
| 08:00:25 | 5779 |
| 08:00:26 | 5777 |
| 08:00:27 | NULL |
| 08:00:28 | NULL |
| 08:00:29 | 5776 |
+----------+-------+
As you can see, there are some points where no data is available (quote is NULL). What I would like to do is a simple step interpolation. This means each NULL value should be updated with the last value available. The only way I managed to do this is with cursors, which is pretty slow due to the large amount of data. I'm basically searching something like this:
UPDATE table AS t1
SET quote = (SELECT quote FROM table AS t2
WHERE t2.tick < t1.tick AND
t2.quote IS NOT NULL
ORDER BY t2.tick DESC
LIMIT 1)
WHERE quote IS NULL
Of course this query will not work, but this is how it should look like.
I would appreciate any ideas on how this can be solved without cursors and temp tables.

Answer 1

This should work:
SET #prev = NULL;
UPDATE ticks
SET quote= #prev := coalesce(quote, #prev)
ORDER BY tick;
BTW the same trick works for reading:
SELECT t.tick, #prev := coalesce(t.quote, #prev)
FROM ticks t
JOIN (SELECT #prev:=NULL) as x -- initializes #prev
ORDER BY tick

Answer 2

The main problem here is reference to main query in subquery t2.tick < t1.tick. Because of this you can't simply wrap the subquery in a another subquery.
If this is one time query and there is not so many data, you can do something like that:
UPDATE `table` AS t1
SET quote = (SELECT quote FROM (SELECT quote, tick FROM `table` AS t2 WHERE t2.quote IS NOT NULL) as t3 WHERE t3.tick < t1.tick ORDER BY t3.tick DESC LIMIT 1)
WHERE quote IS NULL
But really, really don't use that as it will be probably to slow. On each null quote, this query selects all data from table table and then from results it gets desired row.

Answer 3

I would create a (temporary) table with the same layout as your table and run the following two queries:
Insert all interpolations into the temp_stock table
INSERT INTO temp_stock (tick, quote)
SELECT s2.tick
, (s1.quote + s3.quote) /2 as quote
FROM stock
INNER JOIN stock s1 ON (s1.tick < s2.tick)
INNER JOIN stock s3 ON (s3.tick > s2.tick)
WHERE s2.quote IS NULL
GROUP BY s2.tick
HAVING s1.tick = MAX(s1.tick), s3.tick = MIN(s3.tick)
Update the stock table with the temp values
UPDATE stock s
INNER JOIN temp_stock ts ON (ts.tick = s.tick) SET s.quote = ts.quote
It does use a temp table (make sure it's a memory table for speed), but it doesn't need a cursor.