Sum different date intervals year over year - mysql

Question

I have a number of stores where I would like to sum the energy consumption so far this year compared with the same period last year. My challenge is that in the current year the stores have different date intervals in terms of delivered data. That means that store A may have data between 01.01.2018 and 20.01.2018, and store B may have data between 01.01.2018 and 28.01.2018. I would like to sum the same date intervals current year versus previous year.
Data looks like this
Store Date Sum
A 01.01.2018 12
A 20.01.2018 11
B 01.01.2018 33
B 28.01.2018 32
But millions of rows and would use these dates as references to get the same sums previous year.
This is my (erroneous) try:
SET #curryear = (SELECT YEAR(MAX(start_date)) FROM energy_data);
SET #maxdate_curryear = (SELECT MAX(start_date) FROM energy_data WHERE
YEAR(start_date) = #curryear);
SET #mindate_curryear = (SELECT MIN(start_date) FROM energy_data WHERE
YEAR(start_date) = #curryear);
-- the same date intervals last year
SET #maxdate_prevyear = (#maxdate_curryear - INTERVAL 1 YEAR);
SET #mindate_prevyear = (#mindate_curryear - INTERVAL 1 YEAR);
-- sums current year
CREATE TABLE t_sum_curr AS
SELECT name as name_curr, sum(kwh) as sum_curr, min(start_date) AS
min_date_curr, max(start_date) AS max_date_curr, count(distinct
start_date) AS ant_timer FROM energy_data WHERE agg_type = 'timesnivå'
AND start_date >= #mindate_curryear and start_date <= #maxdate_curryear GROUP BY NAME;
-- also seems fair, the same dates one year ago, figured I should find those first and in the next query use that to sum each stores between those date intervals
CREATE TABLE t_sum_prev AS
SELECT name_curr as name_curr2, (min_date_curr - INTERVAL 1 YEAR) AS
min_date_prev, (max_date_curr - INTERVAL 1 YEAR) as max_date_prev FROM
t_sum_curr;
-- getting into trouble!
CREATE TABLE the_results AS
SELECT name, start_date, sum(kwh) as sum_prev from energy_data where
agg_type = 'timesnivå' and
start_date >= #mindate_prevyear and start_date <=
#maxdate_prevyear group by name having start_date BETWEEN (SELECT
min_date_prev from t_sum_prev) AND
(SELECT max_date_prev from t_sum_prev);
`
This last query just tells me that my sub query returns more than 1 row and throws an error message.

Answer 1

I assume what you have is a list of energy consumption figures, where bills or readings have been taken at irregular times, so the consumption covers irregular periods.
The basic approach you need to take is to regularise the consumption periods - by establishing which days each periods covers, and then breaking each reading down into as many days as it covers, and the consumption for each day being a daily average of the period.
I'm assuming the consumption periods are entirely sequential (as a bill or reading normally would be), and not overlapping.
Because of the volume of rows involved (you say millions even in its current form), you might not want to leave the data in daily form - it might suffice to regroup them into regular weekly, monthly, or quarterly periods, depending on what level of granularity you require for comparison.
Once you have your regular periods, comparison will be as easy as cake.
If this is part of a report that will be run on an ongoing basis, you'd probably want to implement some logic that calculates a "regularised consumption" incrementally and on a scheduled basis and stores it in a summary table, with appropriate columns and indexes, so that you aren't having to process many millions of historical rows each time the report is run.
Trying to work around the irregular periods (if indeed it can be done) with fancy joins and on-the-fly averages, rather than tackling them head on, will likely lead to very difficult logic, and particularly on a data set of this size, dire performance.
EDIT: from the comments below.
#Alexander, I've knocked together an example of a query. I haven't tested it and I've written it all in a text editor, so excuse any small syntax errors. What I've come up with seems a bit complex (more complex than I imagined when I began), but I'm also a little bit tired, so I'm not sure whether it could be simplified further.
The only point I would make is that the performance of this query (or any such query), because of the nature of what it has to do in traversing date ranges, is likely to be appalling on a table with millions of rows. I stand by my earlier remarks, that proper indexing of the source data will be crucial, and summarising the source data into a larger granularity will massively aid performance (at the expense of a one-off hit to summarise it). Even daily granularity, will reduce the number of rows by a factor of 24!
WITH energy_data_ext AS
(
SELECT
ed.name AS store_name
,YEAR(ed.start_date) AS reading_year
,ed.start_date AS reading_date
,ed.kwh AS reading_kwh
FROM
energy_data AS ed
)
,available_stores AS
(
SELECT ede.store_name
FROM energy_data_ext AS ede
GROUP BY ede.store_name
)
,current_reading_yr_per_store AS
(
SELECT
ede.store_name
,MAX(ede.reading_year) AS current_reading_year
FROM
energy_data_ext AS ede
GROUP BY
ede.store_name
)
,latest_reading_ranges_per_year AS
(
SELECT
ede.store_name
,ede.reading_year
,MAX(ede.start_date) AS latest_reading_date_of_yr
FROM
energy_data_ext AS ede
GROUP BY
ede.store_name
,ede.reading_year
)
,store_reading_ranges AS
(
SELECT
avs.store_name
,lryps.current_reading_year
,lyrr.latest_reading_date_of_yr AS current_year_latest_reading_date
,(lryps.current_reading_year - 1) AS prev_reading_year
,(lyrr.latest_reading_date_of_yr - INTERVAL 1 YEAR) AS prev_year_latest_reading_date
FROM
available_stores AS avs
LEFT JOIN
current_reading_yr_per_store AS lryps
ON (lryps.store_name = avs.store_name)
LEFT JOIN
latest_reading_ranges_per_year AS lyrr
ON (lyrr.store_name = avs.store_name)
AND (lyrr.reading_year = lryps.current_reading_year)
)
--at this stage, we should have all the calculations we need to
--establish the range for the latest year, and the range for the year prior to that
,current_year_consumption AS
(
SELECT
avs.store_name
SUM(cyed.reading_kwh) AS latest_year_kwh
FROM
available_stores AS avs
LEFT JOIN
store_reading_ranges AS srs
ON (srs.store_name = avs.store_name)
LEFT JOIN
energy_data_ext AS cyed
ON (cyed.reading_year = srs.current_reading_year)
AND (cyed.reading_date <= srs.current_year_latest_reading_date)
GROUP BY
avs.store_name
)
,prev_year_consumption AS
(
SELECT
avs.store_name
SUM(pyed.reading_kwh) AS prev_year_kwh
FROM
available_stores AS avs
LEFT JOIN
store_reading_ranges AS srs
ON (srs.store_name = avs.store_name)
LEFT JOIN
energy_data_ext AS pyed
ON (pyed.reading_year = srs.prev_reading_year)
AND (pyed.reading_date <= srs.prev_year_latest_reading_date)
GROUP BY
avs.store_name
)
SELECT
avs.store_name
,srs.current_reading_year
,srs.current_year_latest_reading_date
,lyc.latest_year_kwh
,srs.prev_reading_year
,srs.prev_year_latest_reading_date
,pyc.prev_year_kwh
FROM
available_stores AS avs
LEFT JOIN
store_reading_ranges AS srs
ON (srs.store_name = avs.store_name)
LEFT JOIN
current_year_consumption AS lyc
ON (lyc.store_name = avs.store_name)
LEFT JOIN
prev_year_consumption AS pyc
ON (pyc.store_name = avs.store_name)