MySQL report -- fill in empty dates - mysql

Question

I am building a query to return daily sales data. My current query returns a table similar to this:
----------------------------------
| DATE | SKU | TOTAL |
----------------------------------
| 2014-11-01 | AV155_A | 209.00 |
| 2014-11-02 | AV155_B | 627.00 |
| 2014-11-04 | AV155_C | 279.00 |
| 2014-11-05 | AV155 | 279.00 |
| 2014-11-08 | AV1556_A | 209.00 |
| 2014-11-09 | AV1556_B | 627.00 |
| 2014-11-10 | AV1556_C | 279.00 |
| 2014-11-12 | AV1556 | 279.00 |
What I would like is a results table that displays every day, even if there are no data points for that particular day. Something like this:
----------------------------------
| DATE | SKU | TOTAL |
----------------------------------
| 2014-11-01 | AV155_A | 209.00 |
| 2014-11-02 | AV155_B | 627.00 |
| 2014-11-03 | | 0 |
| 2014-11-04 | AV155_C | 279.00 |
| 2014-11-05 | AV155 | 279.00 |
| 2014-11-06 | | 0 |
| 2014-11-07 | | 0 |
| 2014-11-08 | AV1556_A | 209.00 |
| 2014-11-09 | AV1556_B | 627.00 |
| 2014-11-10 | AV1556_C | 279.00 |
| 2014-11-11 | | 0 |
| 2014-11-12 | AV1556 | 279.00 |
The query I currently have looks like this:
select
DATE_FORMAT(created_on, '%m-%d-%Y') as date,
sku,
SUM(price) as total
FROM order_items
WHERE created_on between FROM_UNIXTIME(1415577600) AND NOW()
GROUP BY MONTH(created_on), DAY(v.created_on), order_item_sku;

Answer 1

You need to use an outer join. The easiest way is if you have a calendar table, but you can make one on the fly:
select c.thedate, oi.sku, sum(price) as total
from (select date('2014-11-01') as thedate union all
date('2014-11-02') as thedate union all
date('2014-11-03') as thedate union all
date('2014-11-04') as thedate union all
date('2014-11-05') as thedate union all
date('2014-11-06') as thedate union all
date('2014-11-07') as thedate union all
date('2014-11-08') as thedate union all
date('2014-11-09') as thedate union all
date('2014-11-10') as thedate union all
date('2014-11-11') as thedate union all
date('2014-11-12') as thedate
) c left join
order_items oi
on c.thedate = date(oi.created_on)
where oi.created_on between FROM_UNIXTIME(1415577600) AND NOW()
group by ci.thedate, oi.sku

Answer 2

Here's an answer that addresses the need for a flexible list of dates. You need to figure out a way to get a virtual table containing all the dates in the appropriate range, and then join them to the summary. Here’s a query that will get the dates in the range.
SELECT mintime + INTERVAL seq.seq DAY AS reportdate
FROM (
SELECT MIN(DATE(created_on)) AS mintime,
MAX(DATE(created_on)) AS maxtime
FROM order_items
WHERE created_on >= starting_time
AND created_on <= NOW()
) AS order_items
JOIN seq_0_to_999 AS seq
ON seq.seq < TIMESTAMPDIFF(DAY,mintime,maxtime)
What’s going on here? Three things.
We have a subquery which determines the first and last day (min and max created_on) we care about reporting.
We apply a time range to that query. I like to avoid using BETWEEN for timestamp ranges because it often gets the ending time wrong in an off-by-one-second error.
We have a table called seq_0_to_999. It contains a sequence of a thousand cardinal numbers: the integers starting at zero. More about this in a moment.
Then, you can join that as a subquery to your aggregate query to get all the dates in the range listed, like so.
select DATE_FORMAT(d.reportdate, '%m-%d-%Y') as date,
sku,
SUM(price) as total
FROM (
SELECT mintime + INTERVAL seq.seq DAY AS reportdate
FROM (
SELECT MIN(DATE(created_on)) AS mintime,
MAX(DATE(created_on)) AS maxtime
FROM order_items
WHERE created_on >= starting_time
AND created_on <= NOW()
) AS order_items
JOIN seq_0_to_999 AS seq
ON seq.seq < TIMESTAMPDIFF(DAY,mintime,maxtime)
) AS d
LEFT JOIN order_items ON d.reportdate = DATE(order_items.created_on)
WHERE created_on >= starting_time
AND created_on <= NOW()
GROUP BY d.reportdate, sku
ORDER BY d.reportdate, sku
It looks like a big nasty hairball of a query. But if you think of it as a sandwich made of various layers of queries, it really isn't that complicated.
It uses LEFT JOIN so it makes sure all the dates in the range are preserved even if there's no corresponding data in your order_items table.
Finally, what about this seq_0_to_999 table? Where do we get those integers starting with zero? The answer is this: we have to arrange to do that; those numbers aren’t built in to MySQL. (They are built into the MySQL fork called MariaDB.) Create a short table with the integers from 0-9 in it, like so:
DROP TABLE IF EXISTS seq_0_to_9;
CREATE TABLE seq_0_to_9 AS
SELECT 0 AS seq UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4
UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9;
Then create a view that joins that table with itself to generate 1000 combinations like this:
DROP VIEW IF EXISTS seq_0_to_999;
CREATE VIEW seq_0_to_999 AS (
SELECT (a.seq + 10 * (b.seq + 10 * c.seq)) AS seq
FROM seq_0_to_9 a
JOIN seq_0_to_9 b
JOIN seq_0_to_9 c
);
I wrote this up in some detail at http://www.plumislandmedia.net/mysql/filling-missing-data-sequences-cardinal-integers/