Using a calendar table to interpolate values across a date range - mysql

Question

BACKGROUND
I am working on a project where I need to capture the 30 day average of values for some id# then use this average to determine if some new value is anomalous. For the purposes of this question, we can assume I only need a 10-day average since the solutions are probably similar. I currently have two tables: history which holds the actual values that I have recorded for specific id# numbers by day but can have some missing days and calendar a date table that has all of the days that I need in my 30 day average.
create table history (
day date not null,
id bigint not null,
category int not null,
value int not null default '0',
primary key (day, id, category),
key category (category)
);
create table calendar (
day date not null primary key
);
I would like to take the existing data that I have in the history table and fill in the missing data by either copying forward a previous value or copying back a forward value. E.g given this data in the history table:
+------------+-----------+----------+-------+
| day | id | category | value |
+------------+-----------+----------+-------+
| 2015-02-19 | 159253663 | 364 | 212 |
| 2015-02-20 | 159253663 | 364 | 211 |
| 2015-02-22 | 159253663 | 364 | 199 |
| 2015-02-23 | 159253663 | 364 | 192 |
| 2015-02-24 | 159253663 | 364 | 213 |
+------------+-----------+--------+---------+
Note: there is no entry for 2015-02-21
I would like to fill in enough data so that I can compute the 10-day average i.e. copy the oldest value (2015-02-19) back to the beginning of my 10-day range then fill in the missing 2015-02-21 value with the previous day's value. The result would be this (stars mark the newly added rows):
+------------+-----------+----------+-------+
| day | id | category | value |
+------------+-----------+----------+-------+
| 2015-02-14 | 159253663 | 364 | 212 | *
| 2015-02-15 | 159253663 | 364 | 212 | *
| 2015-02-16 | 159253663 | 364 | 212 | *
| 2015-02-17 | 159253663 | 364 | 212 | *
| 2015-02-18 | 159253663 | 364 | 212 | *
| 2015-02-19 | 159253663 | 364 | 212 |
| 2015-02-20 | 159253663 | 364 | 211 |
| 2015-02-21 | 159253663 | 364 | 211 | *
| 2015-02-22 | 159253663 | 364 | 199 |
| 2015-02-23 | 159253663 | 364 | 192 |
| 2015-02-24 | 159253663 | 364 | 213 |
+------------+-----------+--------+---------+
ATTEMPT
My initial thought was to left join to a calendar table that has the date ranges I need, when I do that I get something like this:
select c.day, h.id, h.value
from calendar c
left join history h using (day)
where c.day between curdate() - interval 10 day and curdate();
+------------+-----------+----------+-----------+
| day | id | category | value |
+------------+-----------+----------+-----------+
| 2015-02-14 | NULL | NULL | NULL |
| 2015-02-15 | NULL | NULL | NULL |
| 2015-02-16 | NULL | NULL | NULL |
| 2015-02-17 | NULL | NULL | NULL |
| 2015-02-18 | NULL | NULL | NULL |
| 2015-02-19 | 159253663 | 364 | 212 |
| 2015-02-19 | 159253690 | 364 | 222 |
| 2015-02-20 | 159253663 | 364 | 211 |
| 2015-02-20 | 159253690 | 364 | 221 |
| 2015-02-21 | NULL | NULL | NULL |
| 2015-02-22 | 159253663 | 364 | 199 |
| 2015-02-22 | 159253690 | 364 | 209 |
| 2015-02-23 | 159253663 | 364 | 192 |
| 2015-02-23 | 159253690 | 364 | 202 |
| 2015-02-24 | 159253663 | 364 | 213 |
| 2015-02-24 | 159253690 | 364 | 213 |
+------------+-----------+----------+-----------+
I am not sure where to proceed from this point, because I need an entry for each day for each distinct id#. This join only returns a single day if they are missing. I am looking for a better approach. I would like to push as much of the work as possible on the MySQL server, but can do some things programmaticaly. Any/all ideas or suggestions are welcome.
Here is a SQLFiddle that has the DDL definitions I am testing with: http://sqlfiddle.com/#!2/cc206/2

Answer 1

The following uses an # variable and in-statement assignments to roll backward the value (and id):
SET #lastval = 0, #lastid = 0;
SELECT c.day, #lastid := COALESCE(h.id,#lastid) id, #lastval := COALESCE(h.value,#lastval) VALUE, h.id id1,h.value v1
FROM (SELECT DISTINCT c.day,h.id FROM history h, calendar c) c
LEFT JOIN history h ON h.day = c.day AND h.id = c.id
WHERE c.day BETWEEN CURDATE() - INTERVAL 10 DAY AND CURDATE()
ORDER BY COALESCE(h.id,#lastid),c.day DESC
The sub-query seems to be necessary, never been too sure why (some do, some don`t).
If it looks like the results are in the wrong order you might have to add :
SET optimizer_switch='block_nested_loop=off';
before the statement as the block nested loop optimisation can mess with the Order mysql uses when collecting the rows.