Fill in the missing hours with insert and recursive left join - mysql

I have a table with temperatures.
Sometimes the message was not received and the information is missing.
I need to fill the missing rows with NULL for every hour.
CREATE TABLE temp_total(
id int(6) NOT NULL PRIMARY KEY,
stamp timestamp NOT NULL,
room_temp decimal(3,1) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
ALTER TABLE temp_total
ADD UNIQUE KEY stamp(stamp),
MODIFY id int(6) NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=1;
INSERT INTO temp_total(stamp, room_temp) VALUES
('2019-07-21 19:00:00', '23.4'),
('2019-07-21 22:00:00', '22.7'),
('2019-07-23 02:00:00', '22.5'),
('2019-07-23 06:00:00', '22.4');
The expected result is an array of 36 rows.
I found this query to work fine.
SELECT stamp INTO #deb FROM temp_total ORDER BY stamp ASC LIMIT 1;
SELECT stamp INTO #fin FROM temp_total ORDER BY stamp DESC LIMIT 1;
WITH RECURSIVE all_hours(dt) AS (
SELECT #deb dt
UNION ALL
SELECT dt + INTERVAL 1 HOUR FROM all_hours
WHERE dt + INTERVAL 1 HOUR < #fin + INTERVAL 1 HOUR
)
-- INSERT IGNORE INTO temp_total(stamp, room_temp)
SELECT d.dt stamp, t.room_temp
FROM all_hours d
LEFT JOIN temp_total t ON t.stamp = d.dt
ORDER BY d.dt;
I want to use the result of SELECT with INSERT but I get this message:
#1064 - You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'INSERT INTO temp_total(stamp, room_temp)
SELECT d.dt stamp, t.room_temp
...' at line 7
When I uncomment the line.
DbFiddle

You are almost there. With a small change in the syntax the query works as expected:
INSERT IGNORE INTO temp_total(stamp, room_temp)
WITH RECURSIVE all_hours(dt) AS (
SELECT #deb dt
UNION ALL
SELECT dt + INTERVAL 1 HOUR FROM all_hours
WHERE dt + INTERVAL 1 HOUR < #fin + INTERVAL 1 HOUR
)
SELECT d.dt stamp, t.room_temp
FROM all_hours d
LEFT JOIN temp_total t ON t.stamp = d.dt
ORDER BY d.dt;
See running example at db<>fiddle.

Related

Show all data in a date range using MYSQL recursive function

I'm trying to get a list of sales for the past 6 months and get 0 values if I have no data for a specific month. So I'm using recursive_all_dates to generate a date range for the past 6 months which works great:
with recursive all_dates(dt) as (
-- anchor
select DATE_SUB(now(), INTERVAL 6 MONTH) dt
union all
-- recursion with stop condition
select dt + interval 1 month from all_dates where dt + interval 1 month <= DATE(now())
)
select DATE_FORMAT(dt, '%Y-%m') as ym from all_dates
This will return:
ym
------
2019-10
2019-11
2019-12
2020-01
2020-02
2020-03
2020-04
Now I want to left join this with my real data:
with recursive all_dates(dt) as (
-- anchor
select DATE_SUB(now(), INTERVAL 6 MONTH) dt
union all
-- recursion with stop condition
select dt + interval 1 month from all_dates where dt + interval 1 month <= now()
)
SELECT
DATE_FORMAT(ad.dt, '%Y-%m') as ym,
sum(profit) as profit
FROM
all_dates as ad
LEFT JOIN organisation_invoices as i
ON
DATE_FORMAT(ad.dt, '%Y-%m') = DATE_FORMAT(i.issue_date, '%Y-%m')
JOIN (
SELECT
invoice_id,
SUM(value) as profit
FROM organisation_invoice_services isrv
GROUP BY invoice_id
) isrv
ON i.id = isrv.invoice_id
WHERE
i.organisation_id = '4b166dbe-d99d-5091-abdd-95b83330ed3a' AND
i.issue_date >= DATE_SUB(NOW(), INTERVAL 6 MONTH)
GROUP BY `ym`
ORDER BY `ym` ASC
But I still only get the populated months:
ym profit
------------------
2019-12 8791
2020-02 302
2020-04 10452
The desired result:
ym profit
------------------
2019-10 0
2019-11 0
2019-12 8791
2020-01 0
2020-02 302
2020-03 0
2020-04 10452
What am I missing?
Edit: Sample data set and fiddle:
CREATE TABLE `organisation_invoices` (
`id` varchar(255) NOT NULL,
`organisation_id` varchar(255) NOT NULL,
`issue_date` date NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `organisation_invoice_services` (
`id` varchar(255) NOT NULL,
`organisation_id` varchar(255) NOT NULL,
`invoice_id` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`qty` float NOT NULL,
`value` float NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
INSERT INTO `organisation_invoices` (id, organisation_id, issue_date)
VALUES ('e11cec69-138f-4e20-88e5-5430b6c8d0a1', '4b166dbe-d99d-5091-abdd-95b83330ed3a', '2020-01-20');
INSERT INTO `organisation_invoice_services` (id, organisation_id, invoice_id, qty, `value`)
VALUES ('fe45dfd67-138f-4e20-88e5-5430b6c8d0a1', '4b166dbe-d99d-5091-abdd-95b83330ed3a', 'e11cec69-138f-4e20-88e5-5430b6c8d0a1', 1, 1000);
https://www.db-fiddle.com/f/dibyQi31CBtr2Cr8vjJA8i/0
You can use the following:
with recursive all_dates(dt) as (
-- anchor
select DATE_SUB(now(), INTERVAL 6 MONTH) dt
union all
-- recursion with stop condition
select dt + interval 1 month from all_dates where dt + interval 1 month <= now()
)
SELECT DATE_FORMAT(ad.dt, '%Y-%m') as ym, IFNULL(sum(profit),0) as profit
FROM all_dates as ad
LEFT JOIN organisation_invoices as i
ON DATE_FORMAT(ad.dt, '%Y-%m') = DATE_FORMAT(i.issue_date, '%Y-%m')
LEFT JOIN (
SELECT
invoice_id,
SUM(value) as profit
FROM organisation_invoice_services isrv
GROUP BY invoice_id
) isrv
ON i.id = isrv.invoice_id
WHERE
(i.organisation_id = '4b166dbe-d99d-5091-abdd-95b83330ed3a' AND
i.issue_date >= DATE_SUB(NOW(), INTERVAL 6 MONTH)) OR i.organisation_id IS NULL
GROUP BY `ym`
ORDER BY `ym` ASC
demo on dbfiddle.uk
Changes:
The conditions on the WHERE clause change the behaviour of your LEFT JOIN. Since you check for a specific organization_id, you only get matches between your month table and data (the LEFT JOIN behaves like a INNER JOIN). You need the following WHERE clause instead:
WHERE (i.organisation_id = '4b166dbe-d99d-5091-abdd-95b83330ed3a' AND
i.issue_date >= DATE_SUB(NOW(), INTERVAL 6 MONTH)) OR i.organisation_id IS NULL
You also have to change the second JOIN to a LEFT JOIN.

Select while instead of While select causing issue

I'm working on a ranged date query, and trying to adjust the rules for the loop but I have a bit of a problem:
Take the following:
DROP PROCEDURE
IF EXISTS test;
CREATE PROCEDURE test ( IN start_date DATE ) BEGIN
DECLARE group_name VARCHAR ( 10 ) DEFAULT 'clientA';
DECLARE service_name VARCHAR ( 10 ) DEFAULT 'serviceA';
WHILE ( start_date < CURDATE( ) && SUBDATE( start_date, INTERVAL - 2 WEEK ) < CURDATE( ) ) DO
SELECT start_date AS 'Start Day', SUBDATE( start_date, INTERVAL - 2 WEEK ) AS 'End Day';
SET start_date = SUBDATE( start_date, INTERVAL - 2 WEEK );
END WHILE;
END;
This selects a start and end date from a starting point up to today:
CALL test ( '2019-08-29' );
Returns 5 results:
08/29 & 09/12
09/12 & 09/26
09/26 & 10/10
10/10 & 10/24
10/24 & 11/7
This is what I want but rather than 5 results. I want each of these as rows in one result. I think the best way to do this is via a sub-query with the inner query running the loop and doing the selects but the outer query serving as a wrapper to constrain them into one set.
I have the following code:
DROP PROCEDURE
IF EXISTS test;
CREATE PROCEDURE test ( IN start_date DATE ) BEGIN
DECLARE group_name VARCHAR ( 10 ) DEFAULT 'clientA';
DECLARE service_name VARCHAR ( 10 ) DEFAULT 'serviceA';
SELECT * FROM (WHILE ( start_date < CURDATE( ) && SUBDATE( start_date, INTERVAL - 2 WEEK ) < CURDATE( ) ) DO
SELECT start_date AS 'Start Day', SUBDATE( start_date, INTERVAL - 2 WEEK ) AS 'End Day';
SET start_date = SUBDATE( start_date, INTERVAL - 2 WEEK );
END WHILE;
)
END;
But that gives me:
1064 - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AS 'Result' FROM (
WHILE ( start_date < CURDATE( ) && SUBDATE( start_date, IN' at line 4
I feel like there's something small wrong with my syntax here but I'm having difficulty understanding exactly what. Any guidance would be great!
you can create a dynamic list of date activity by using MySQL # variables and joining to any table that has as many rows as you expect in result set... ex: if you need 5, 10 or 1000 records in the dynamic result.
select
-- whatever latest date is BECOMES the Begin Date
#beginDT BeginDate,
-- now, add 2 weeks to the #beginDT variable and save as the END Date
#beginDT := date_add( #beginDT, interval 2 week ) EndDate
from
-- pick any table that has as many 2-week cycles as you expect.
-- ex: if you wanted 1 yr, you would need any table with 26 or 27 records
AnyTableWithManyRecords,
-- start the query with your starting date, alias sqlvars is just place-holder
-- and will only prepare the variable and be one row for rest of query
( select #beginDT := '2019-08-29' ) sqlvars
having
-- having will stop until your maximum date of interest
BeginDate < curdate()
-- but limit to 100 so you don't query against table of millions of records.
-- how many records do you REALLY need to go through... again, 26 biweekly = 1 year
-- this limit of 100 would allow for almost 4 yrs worth
limit 100;
Then, if you wanted data from some other table, you could join to the above result set as its own such as
select
SOT.WhateverColumns
from
( above query ) MyDates
JOIN SomeOtherTable SOT
on MyDates.BeginDate <= SOT.SomeDate
AND SOT.SomeDate < MyDates.EndDate

Rewrite sql query to pad empty month rows

I have this query i use to get statistics of blogs in our own tracking system.
I use union select over 2 tables as we daily aggregate data in 1 table and keeps todays data in another table.
I want to have the last 10 months of traffic show.. This query does that, but of there is no traffic in a specific month that row is not in the result.
I have previously used a calendar table in mysql to join against to at avoid that, but im simply not skilled enoght to rewrite this query to join against that calendar table.
The calendart table has 1 field called "datefield" which i date format YYY-MM-DD
This is the current query i use
SELECT FORMAT(SUM(`count`),0) as `count`, DATE(`date`) as `date`
FROM
(
SELECT count(distinct(uniq_id)) as `count`, `timestamp` as `date`
FROM tracking
WHERE `timestamp` > now() - INTERVAL 1 DAY AND target_bid = 92
group by `datestamp`
UNION ALL
select sum(`count`),`datestamp` as `date`
from aggregate_visits
where `datestamp` > now() - interval 10 month
and target_bid = 92
group by `datestamp`
) a
GROUP BY MONTH(date)
Something like this?
select sum(COALESCE(t.`count`,0)),s.date as `date`
from DateTable s
LEFT JOIN (SELECT * FROM aggregate_visits
where `datestamp` > now() - interval 10 month
and target_bid = 92) t
ON(s.date = t.datestamp)
group by s.date

GROUP BY MONTH() hide result

I'm trying to count how many result in each month.
This is my query :
SELECT
COUNT(*) as nb,
CONCAT(MONTH(t.date),0x3a,YEAR(t.date)) as period
FROM table1 t
WHERE t.criteria = 'value'
GROUP BY MONTH(t.date)
ORDER BY YEAR(t.date)
My Result:
nb period
---------------
7 6:2009
46 8:2009
2 10:2009
1 11:2009
14 1:2009
9 9:2010
161 7:2010
5 2:2010
88 3:2010
28 4:2010
4 5:2011
2 12:2011
The problem is, I'm sure that I've result between 5:2011 & 12:2011 , and each other period
since 2009 ... :/
This is a problem of my request or mysql configuration ?
Thx a lot
You have to group by both the year and the month. Otherwise your April 2012 rows are grouped with April 2011 (and April 2010 ...) rows as well.
SELECT
COUNT(*) AS nb,
CONCAT(MONTH(t.date), ':', YEAR(t.date)) AS period
FROM table1 AS t
WHERE t.criteria = 'value'
GROUP BY YEAR(t.date)
, MONTH(t.date) ;
(and is there a reason you used 0x3a and not ':'?)
You could also use some other DATE and TIME functions of MySQL so there are fewer functions calls per row and probably a more efficient query:
SELECT
COUNT(*) AS nb,
DATE_FORMAT(t.date, '%m:%Y') AS period
FROM table1 AS t
WHERE t.criteria = 'value'
GROUP BY EXTRACT( YEAR_MONTH FROM t.date) ;
For several queries, it's useful to have a permanent Calendar table in your database (with all dates or all year-months) or even several Calendar tables. Example:
CREATE TABLE CalendarYear
( Year SMALLINT UNSIGNED NOT NULL
, PRIMARY KEY (Year)
) ENGINE = InnoDB ;
INSERT INTO CalendarYear
(Year)
VALUES
(1900), (1901), ..., (2099) ;
CREATE TABLE CalendarMonth
( Month TINYINT UNSIGNED NOT NULL
, PRIMARY KEY (Month)
) ENGINE = InnoDB ;
INSERT INTO CalendarMonth
(Month)
VALUES
(1), (2), ..., (12) ;
Those can also help us make the one we'll need here:
CREATE TABLE CalendarYearMonth
( Year SMALLINT UNSIGNED NOT NULL
, Month TINYINT UNSIGNED NOT NULL
, FirstDay DATE NOT NULL
, NextMonth_FirstDay DATE NOT NULL
, PRIMARY KEY (Year, Month)
) ENGINE = InnoDB ;
INSERT INTO CalendarYearMonth
(Year, Month, FirstDay, NextMonth_FirstDay)
SELECT
y.Year
, m.Month
, MAKEDATE(y.Year, 1) + INTERVAL (m.Month-1) MONTH
, MAKEDATE(y.Year, 1) + INTERVAL (m.Month) MONTH
FROM
CalendarYear AS y
CROSS JOIN
CalendarMonth AS m ;
Then you can use the Calendar tables to write more complex queries, like the variation you want (with missing months) and probably more efficiently. Tested in SQL-Fiddle:
SELECT
COUNT(t.date) AS nb,
CONCAT(cal.Month, ':', cal.Year) AS period
FROM
CalendarYearMonth AS cal
JOIN
( SELECT
YEAR(MIN(date)) AS min_year
, MONTH(MIN(date)) AS min_month
, YEAR(MAX(date)) AS max_year
, MONTH(MAX(date)) AS max_month
FROM table1
WHERE criteria = 'value'
) AS mm
ON (cal.Year, cal.Month) >= (mm.min_year, mm.min_month)
AND (cal.Year, cal.Month) <= (mm.max_year, mm.max_month)
LEFT JOIN
table1 AS t
ON t.criteria = 'value'
AND t.date >= cal.FirstDay
AND t.date < cal.NextMonth_FirstDay
GROUP BY
cal.Year, cal.Month ;
You must also GROUP BY the year:
GROUP BY MONTH(t.date), YEAR(t.date)
Your original query uses YEAR(t.date) in the SELECT clause outside of any aggregate function without grouping by it -- as a result, you get exactly 12 groups (one for each possible month) and for each group (that possibly contains dates across many years) a "random" year is chosen by MySql for selection. Strictly speaking, this is meaningless and the query should never have been allowed to execute. But MySql... sigh.

MySQL Specialists: Delete all rows older than x days but not the latest one

first of all, this is the query which creates the "player history"
it can be executed as often as you want and it will only create new history rows for the players if there is no history row for yesterday or if the values changed since the latest history entry in the past.
INSERT INTO `player_history` (`player_id`, `date`, `races`, `maps`, `playtime`, `points`)
SELECT `p`.`id`, DATE_SUB(NOW(), INTERVAL 1 DAY), `p`.`races`, `p`.`maps`, `p`.`playtime`, `p`.`points`
FROM `player` `p`
WHERE `p`.`playtime` IS NOT NULL
AND `p`.`playtime` > 0
AND (
SELECT `player_id`
FROM `player_history`^
WHERE `player_id` = `p`.`id`
AND (
`date` = DATE_SUB(NOW(), INTERVAL 1 DAY)
OR (
`date` < DATE_SUB(NOW(), INTERVAL 1 DAY)
AND `races` = `p`.`races`
AND `points` = `p`.`points`
AND `maps` = `p`.`maps`
AND `playtime` = `p`.`playtime`
)
)
ORDER BY `date` DESC
LIMIT 1
) IS NULL;
now the problem is i also want to cleanup the history table using a single query. this already selects all history entries older than 10 days but the latest. but i cant just like do DELETE instead of SELECT *.
SELECT *
FROM `player_history` `ph`
WHERE `date` < DATE_SUB(NOW(), INTERVAL 10 DAY)
AND `date` != (SELECT `date`
FROM `player_history`
WHERE `player_id` = `ph`.`player_id`
ORDER BY `date` DESC
LIMIT 1);
so is tehre a way to do what i want using a single delete query?
Your query looks right in my eyes but you don't have the interval in the subquery.
I would do this:
DELETE FROM player_history
WHERE date < DATE_SUB(NOW(), INTERVAL 10 DAY)
AND date != (
SELECT MAX(date) FROM player_history
WHERE date < DATE_SUB(NOW(), INTERVAL 10 DAY)
)
What's the error message from mysql?
Probably you can't do this in a single query because the documentation states:
Currently, you cannot delete from a table and select from the same table in a subquery.
As a workaround you could select the ids of the rows that have to be deleted into a temporary table and then use a multi-table delete statement to delete the records from the original table.