Filling date gaps with most recent value - mysql

Question

I'm developing a script at my company that will pull information from our SCM about source code activity, such as the number of lines changed, for a given product over time. All the changes for a given product that occur within the same day are combined into a single record in a mySQL table, something like this:
+------------+-------+------+
| date | prod | line |
+------------+-------+------+
| 2011-11-25 | prod2 | 471 |
| 2011-11-28 | prod2 | 389 |
+------------+-------+------+
I then replicate the table with the cumulative results using an inner join and summation:
+------------+-------+------+
| date | prod | line |
+------------+-------+------+
| 2011-11-25 | prod2 | 471 |
| 2011-11-28 | prod2 | 860 |
+------------+-------+------+
Now, I want to create a table that has one record for every day per product. I've been able to do that by joining with a calendar table. However, where new records are created, the line field should be populated with the most recent cumulative value for that product, rather than some hardcoded default like NULL or 0:
+------------+-------+------+
| date | prod | line |
+------------+-------+------+
| 2011-11-25 | prod2 | 471 |
| 2011-11-26 | prod2 | 471 |
| 2011-11-27 | prod2 | 471 |
| 2011-11-28 | prod2 | 860 |
+------------+-------+------+
I've solved this problem in two unsatisfying ways so far:
Fill in the date gaps first, then compute the cumulative sum
Loop over every element of the final table, saving the most recent non-null elements in a #user variable.
The first solution became hugely inefficient once my table grew large enough. The second solution gets the job done, but I've been trying to find a more elegant solution. Here is the code that produces the table with NULLs:
INSERT INTO final SELECT d.date,f.prod,p.line
FROM calendar AS d
CROSS JOIN
(SELECT DISTINCT prod FROM cumulative) AS f
LEFT JOIN cumulative AS p USING (date,prod) ;
Any ideas? I'm using MySQL.

Answer 1

Seems like the most sensible thing to do would be to store one row per day, with a zero if there were no lines changed. That would eliminate the need for a join on the calendar table.
So instead of your source table looking like this
+------------+-------+------+
| date | prod | line |
+------------+-------+------+
| 2011-11-25 | prod2 | 471 |
| 2011-11-28 | prod2 | 389 |
+------------+-------+------+
it would look like this.
+------------+-------+------+
| date | prod | line |
+------------+-------+------+
| 2011-11-25 | prod2 | 471 |
| 2011-11-26 | prod2 | 0 |
| 2011-11-27 | prod2 | 0 |
| 2011-11-28 | prod2 | 389 |
+------------+-------+------+
As for the running sum itself, your report writer might be able to do this faster than SQL. If MySQL supported windowing functions, you'd just write something like
select date, prod,
sum(line) over (partition by prod order by date)
from prod
although, even then, your report writer might be faster.
On platforms that don't support windowing functions, you just need a sum in a subquery.
select p1.prod, p1.date,
(select sum(line) from prod
where prod = p1.prod and date <= p1.date) as num_lines
from prod p1
order by p1.prod, p1.date