MySQL Field Data Parsing - mysql

Started here, I was asked to create a new post in order to flesh it out which I'll try to do here but I'm afraid it's rather wordy. I created a test table with what seems to be a good cross-section of the data that needs to be parsed. Using this query from here to which I added some additional fields that are needed, this is the query now.
SELECT ID, PartNo,
GROUP_CONCAT(
REPLACE(t.part, '-', CONCAT(' ', LEFT(t.part, 2)))
ORDER BY t.ModelList
SEPARATOR ' '
) Models
FROM (
SELECT t.ID, t.PartNo, t.Models, p.ModelList,
REPLACE(REPLACE(
SUBSTRING_INDEX(t.Models, ';', p.ModelList),
SUBSTRING_INDEX(t.Models, ';', p.ModelList - 1),
''
), ';', '') part
FROM parts_listing t CROSS JOIN (
SELECT 1 ModelList UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL
SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL
SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9
) p
WHERE REPLACE(REPLACE(Models, '-', ''), ';', '') REGEXP '^[0-9]*$'
) t
WHERE t.part <> ''
GROUP BY t.ID, t.PartNo, t.Models
ORDER BY t.PartNo, t.Models
For the most basic parts, the Models data looks like this:
2206-13-26-33; 2302-06-13-32-33
. . . and this is the part that I am working on now. The Models column also contains ALL 23RD etc that will also need to be parsed somehow but for now I am concerned only with the data as shown above and I need to parse it to something like:
2206 2213 2226 2233 2302 2306 2313 2332 2333
The query above, however, seems to be repeating the first bit so it comes out as:
2206 2213 2226 2233 2202 2206 2213 2232 2233
It gives the result in one chunk but it would also be okay if each result was on it's own line as ultimately it will be dumped into another table and that is the way it will be by then.
It also gives too few results. The actual table contains over 15,000 rows but this is fetching only a small fraction and in the data sample below, it should fetch about 20 rows but actually fetches only 6 although the fiddle, which has a very limited data set, does indeed give the proper results. Running the the fiddle query against my own data fetches only 555 rows.
A cross-section sample of data can be created with this:
DROP TABLE IF EXISTS `parts_listing`;
CREATE TABLE IF NOT EXISTS `parts_listing` (
`ID` smallint(5) unsigned NOT NULL AUTO_INCREMENT,
`GroupNumber` varchar(20) DEFAULT NULL,
`PartNo` varchar(30) DEFAULT NULL,
`Models` varchar(255) DEFAULT NULL,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=34 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT;
INSERT INTO `parts_listing` (`ID`, `GroupNumber`, `PartNo`, `Models`) VALUES
(1, '0.000', '391906', '2201-02-11-20-22-32-40; 2301'),
(2, '0.000', '391907', '2206-26-33'),
(3, '0.000', '399391', '2213'),
(4, '0.000', '415209', '2302-06-32-33'),
(5, '0.000', '415245', '2313'),
(6, '0.000', '437273', 'ALL 24TH; 25TH; 26TH'),
(7, '0.000', '436803', '2631 (BODY 2678); 5431 (BODY 5478)'),
(8, '0.000', '448611', 'ALL 54TH'),
(9, '0.0015', '372174', 'ALL 22ND; 23RD'),
(10, '0.0015', '399324', '2201-02-11-20-22-32-40; 2301'),
(11, '0.0015', '399324', '2206-13-26-33; 2302-06-13-32-33'),
(12, '0.0015', '372146', 'ALL 22ND; 23RD'),
(13, '0.0015', 'G179359', '2201-02-11-20-22-32-40; 2301'),
(14, '0.0015', 'G179559', '2206-13-26-33; 2302-06-13-32-33'),
(15, '0.0015', '422024', '2201-02-11-20-22-32-40; 2301'),
(16, '0.0015', '422024', '2206-13-26-33; 2302-06-13-32-33'),
(17, '0.0016', '378664', 'ALL 22ND; 23RD'),
(18, '0.0016', 'G178532', 'ALL 22ND; 23RD'),
(19, '0.0016', 'G123197', 'ALL 22ND; 23RD'),
(20, '30.793', '448421', 'ALL 54TH'),
(21, '30.7932', '422762', '2462-65-92-95-98; 2562-65-92-95; 2662-65-92-95'),
(22, '30.7932', '448902', '5462-65-67-82-85-92-95-97'),
(23, '30.7938', '399941', 'ALL 22ND; 23RD'),
(24, '30.7938', '425809', 'ALL 24TH; 25TH; 26TH'),
(25, '30.7938', '447799', 'ALL 54TH'),
(26, '30.79383', '399927', 'ALL 22ND; 23RD; 24TH; 25TH; 26TH'),
(27, '30.79396', 'PA437550', '2562-65-77-79; 2677-79'),
(28, '30.79398', '422470', '2452-72; 2552-72'),
(29, '30.79398', '422471', '2452-72; 2552-72'),
(30, '30.79398', '422553', 'ALL 24TH; 25TH'),
(31, '30.8007', '425517', '2413-52-62-65-67-72-92-95-98; 2552-62-65-72-77-92-95; 2650-51-52-62-65-72-77-92-95-97; 5450-51-52-62-65-67-72-77-82-85-92-95-97'),
(32, '30.8007', '425920', 'RHD, 2462-65-92-95; 2562-65-92-95; 2662-65-92-95-97; 5462-65-82-85-92-95-97'),
(33, '30.8044', 'G129662', '2293');


I think you need a more sophisticated way to verify that the Models column values ​​are correct due to spaces after semicolons. In addition, if you need to process the Models column values that contain more than ten parts, you will need a longer sequence of numbers. I use the INFORMATION_SCHEMA COLUMNS table for this. You can also create an auxiliary table for this.
SELECT
CONCAT(
IF(N.n = 0, '', LEFT(t.part, 2)),
SUBSTRING_INDEX(
SUBSTRING_INDEX(t.part, '-', N.n + 1), '-', -1
)
) AS part, t.isHD, t.ID, t.GroupNumber, t.PartNo, t.Models
FROM Numbers AS N CROSS JOIN (
SELECT
SUBSTRING_INDEX(
SUBSTRING_INDEX(t.part, ' ', N.n + 1), ' ', -1
) AS part, t.isHD, t.ID, t.GroupNumber, t.PartNo, t.Models
FROM Numbers AS N CROSS JOIN (
SELECT
SUBSTRING_INDEX(
SUBSTRING_INDEX(t.part, ';', N.n + 1), ';', -1
) AS part, t.isHD, t.ID, t.GroupNumber, t.PartNo, t.Models
FROM Numbers AS N CROSS JOIN (
SELECT
REPLACE(
SUBSTRING_INDEX(
SUBSTRING_INDEX(t.part, 'LHD', N.n + 1), 'LHD', -1
), ',', ''
) AS part, IF(N.n > 0, 1, t.isHD) AS isHD,
t.ID, t.GroupNumber, t.PartNo, t.Models
FROM Numbers AS N CROSS JOIN (
SELECT
SUBSTRING_INDEX(
SUBSTRING_INDEX(p.Models, 'RHD', N.n + 1), 'RHD', -1
) AS part, IF(N.n > 0, 2, 0) AS isHD,
p.ID, p.GroupNumber, p.PartNo, p.Models
FROM Numbers AS N CROSS JOIN parts_listing AS p
WHERE p.Models REGEXP '[[:<:]]([LR]HD,? +)?[0-9]+(-[0-9]+)*[[:>:]]' AND
N.n <= (LENGTH(p.Models) - LENGTH(REPLACE(p.Models, 'RHD', ''))) / 3
) AS t WHERE N.n <= (LENGTH(t.part) - LENGTH(REPLACE(t.part, 'LHD', ''))) / 3
) AS t WHERE N.n <= LENGTH(t.part) - LENGTH(REPLACE(t.part, ';', ''))
) AS t WHERE N.n <= LENGTH(t.part) - LENGTH(REPLACE(t.part, ' ', ''))
) AS t WHERE t.part REGEXP '^[0-9]+(-[0-9]+)*$' AND
N.n <= LENGTH(t.part) - LENGTH(REPLACE(t.part, '-', ''));
Partial output (29 rows):
+------+------+----+-------------+---------+--------------------------------------+
| part | isHD | ID | GroupNumber | PartNo | Models |
+------+------+----+-------------+---------+--------------------------------------+
| 2201 | 0 | 1 | 0.000 | 391906 | 2201-02-11-20-22-32-40; 2301 |
| 2202 | 0 | 1 | 0.000 | 391906 | 2201-02-11-20-22-32-40; 2301 |
| 2211 | 0 | 1 | 0.000 | 391906 | 2201-02-11-20-22-32-40; 2301 |
| 2220 | 0 | 1 | 0.000 | 391906 | 2201-02-11-20-22-32-40; 2301 |
| 2222 | 0 | 1 | 0.000 | 391906 | 2201-02-11-20-22-32-40; 2301 |
| 2232 | 0 | 1 | 0.000 | 391906 | 2201-02-11-20-22-32-40; 2301 |
| 2240 | 0 | 1 | 0.000 | 391906 | 2201-02-11-20-22-32-40; 2301 |
| 2301 | 0 | 1 | 0.000 | 391906 | 2201-02-11-20-22-32-40; 2301 |
| 2213 | 1 | 3 | 0.000 | 399391 | LHD 2213 |
| 2313 | 2 | 5 | 0.000 | 415245 | RHD, 2313 |
| 2631 | 0 | 7 | 0.000 | 436803 | 2631 (BODY 2678); 5431 (BODY 5478) |
| 5431 | 0 | 7 | 0.000 | 436803 | 2631 (BODY 2678); 5431 (BODY 5478) |
| 2206 | 1 | 14 | 0.0015 | G179559 | LHD, 2206-13-26-33; 2302-06-13-32-33 |
| 2213 | 1 | 14 | 0.0015 | G179559 | LHD, 2206-13-26-33; 2302-06-13-32-33 |
| 2226 | 1 | 14 | 0.0015 | G179559 | LHD, 2206-13-26-33; 2302-06-13-32-33 |
| 2233 | 1 | 14 | 0.0015 | G179559 | LHD, 2206-13-26-33; 2302-06-13-32-33 |
| 2302 | 1 | 14 | 0.0015 | G179559 | LHD, 2206-13-26-33; 2302-06-13-32-33 |
| 2306 | 1 | 14 | 0.0015 | G179559 | LHD, 2206-13-26-33; 2302-06-13-32-33 |
| 2313 | 1 | 14 | 0.0015 | G179559 | LHD, 2206-13-26-33; 2302-06-13-32-33 |
| 2332 | 1 | 14 | 0.0015 | G179559 | LHD, 2206-13-26-33; 2302-06-13-32-33 |
| 2333 | 1 | 14 | 0.0015 | G179559 | LHD, 2206-13-26-33; 2302-06-13-32-33 |
| 2201 | 0 | 15 | 0.0015 | 422024 | 2201-02-11-20-22-32-40; RHD, 2301 |
| 2202 | 0 | 15 | 0.0015 | 422024 | 2201-02-11-20-22-32-40; RHD, 2301 |
| 2211 | 0 | 15 | 0.0015 | 422024 | 2201-02-11-20-22-32-40; RHD, 2301 |
| 2220 | 0 | 15 | 0.0015 | 422024 | 2201-02-11-20-22-32-40; RHD, 2301 |
| 2222 | 0 | 15 | 0.0015 | 422024 | 2201-02-11-20-22-32-40; RHD, 2301 |
| 2232 | 0 | 15 | 0.0015 | 422024 | 2201-02-11-20-22-32-40; RHD, 2301 |
| 2240 | 0 | 15 | 0.0015 | 422024 | 2201-02-11-20-22-32-40; RHD, 2301 |
| 2301 | 2 | 15 | 0.0015 | 422024 | 2201-02-11-20-22-32-40; RHD, 2301 |
+------+------+----+-------------+---------+--------------------------------------+
Demo on DB Fiddle.


You can try this. I changed the parts where You split the Models column.
I split them into different rows, and let the code from forpas go over it and join them again together. rows with text in it aren't touched, because you didn't specify nothing.
select Id,GroupNumber,PartNo,
group_concat(
replace(t.part, '-', concat(' ', left(t.part, 2)))
order by t.partno, t.part ASC
separator ' '
) Models
from
(SELECT
Id,GroupNumber,PartNo,
TRIM(SUBSTRING_INDEX(SUBSTRING_INDEX(models, ';', n.digit+1), ';', -1)) part
FROM
( SElECT Id,GroupNumber,PartNo,
REPLACE(REPLACE(REPLACE(Models, ')', ''), 'RHD,', ''), '(BODY', ';') Models
FROM
parts_listing ) p_l
INNER JOIN
(SELECT 0 digit UNION ALL select 1 union all select 2 union all select 3 union all
select 4 union all select 5 union all select 6 union all
select 7 union all select 8 union all select 9) n
ON LENGTH(REPLACE(Models, ';' , '')) <= LENGTH(Models)-n.digit
Where REPLACE(REPLACE(REPLACE(Models, '-', ''), ' ', ''), ';', '') REGEXP '^[0-9]+$'
ORDER BY
Id,
n.digit,part) t
Group by Id,GroupNumber,PartNo
;
Gets you
Id GroupNumber PartNo Models
1 0.000 391906 2201 2202 2211 2220 2222 2232 2240 2301
2 0.000 391907 2206 2226 2233
3 0.000 399391 2213
4 0.000 415209 2302 2306 2332 2333
5 0.000 415245 2313
7 0.000 436803 2631 2678 5431 5478
10 0.0015 399324 2201 2202 2211 2220 2222 2232 2240 2301
11 0.0015 399324 2206 2213 2226 2233 2302 2306 2313 2332 2333
13 0.0015 G179359 2201 2202 2211 2220 2222 2232 2240 2301
14 0.0015 G179559 2206 2213 2226 2233 2302 2306 2313 2332 2333
15 0.0015 422024 2201 2202 2211 2220 2222 2232 2240 2301
16 0.0015 422024 2206 2213 2226 2233 2302 2306 2313 2332 2333
21 30.7932 422762 2462 2465 2492 2495 2498 2562 2565 2592 2595 2662 2665 2692 2695
22 30.7932 448902 5462 5465 5467 5482 5485 5492 5495 5497
27 30.79396 PA437550 2562 2565 2577 2579 2677 2679
28 30.79398 422470 2452 2472 2552 2572
29 30.79398 422471 2452 2472 2552 2572
31 30.8007 425517 2413 2452 2462 2465 2467 2472 2492 2495 2498 2552 2562 2565 2572 2577 2592 2595 2650 2651 2652 2662 2665 2672 2677 2692 2695 2697 5450 5451 5452 5462 5465 5467 5472 5477 5482 5485 5492 5495 5497
32 30.8007 425920 2462 2465 2492 2495 2562 2565 2592 2595 2662 2665 2692 2695 2697 5462 5465 5482 5485 5492 5495 5497
33 30.8044 G129662 2293
Dbfiddle example

Related

Query Group where Timediff is less than 1 minute

I am trying to figure out how to group a query result where the final Grouping should happen where the time difference is less than let's say one minute.
I have watermeter that logs my water usage and I am trying to group the results so that the graphs will make more sense. My sql queries for grouping the water usage per Year, Month, Day and hour are perfect, but then I would like to drill down to where the final result shows me a grouping where as an example I water the grass.
My Table Structure looks like:
id liter total_liters date time dater
9 3 184 2020/12/06 16:14:58 2020/12/06 16:14
10 1 185 2020/12/06 16:15:04 2020/12/06 16:15
11 3 188 2020/12/06 16:26:49 2020/12/06 16:26
12 2 190 2020/12/06 16:26:55 2020/12/06 16:26
13 2 192 2020/12/06 16:27:01 2020/12/06 16:27
14 1 193 2020/12/06 17:32:16 2020/12/06 17:32
15 1 194 2020/12/06 17:32:22 2020/12/06 17:32
16 1 195 2020/12/06 17:32:28 2020/12/06 17:32
17 1 196 2020/12/06 17:32:35 2020/12/06 17:32
18 1 197 2020/12/06 17:32:41 2020/12/06 17:32
19 1 198 2020/12/06 17:32:47 2020/12/06 17:32
20 1 199 2020/12/06 17:32:53 2020/12/06 17:32
21 1 200 2020/12/06 17:32:59 2020/12/06 17:32
22 1 201 2020/12/06 17:35:05 2020/12/06 17:35
23 1 202 2020/12/06 17:35:17 2020/12/06 17:35
24 1 203 2020/12/06 17:35:23 2020/12/06 17:35
25 1 204 2020/12/06 17:35:29 2020/12/06 17:35
26 1 205 2020/12/06 17:35:41 2020/12/06 17:35
27 1 206 2020/12/06 17:43:05 2020/12/06 17:43
28 3 209 2020/12/06 17:43:11 2020/12/06 17:43
29 2 211 2020/12/06 17:43:17 2020/12/06 17:43
30 2 213 2020/12/06 17:43:23 2020/12/06 17:43
31 2 215 2020/12/06 17:43:29 2020/12/06 17:43
32 3 218 2020/12/06 17:43:36 2020/12/06 17:43
33 2 220 2020/12/06 17:43:42 2020/12/06 17:43
And my current query looks like:
SELECT DATE_FORMAT(dater,'%H:%i') AS dater,
YEAR(dater),
MONTHNAME(dater),
DAY(dater),
HOUR(dater),
MINUTE(dater),
SUM(liter) as liter
FROM watermeter
WHERE date LIKE '2020-12-08'
GROUP BY YEAR(date), MONTHNAME(date), DAY(dater), HOUR(dater), MINUTE(dater)
ORDER BY id ASC`
The result should be to sum the Liters together by grouping them by Year then Month then Day then Hour and then it should group the results where the time difference is less than 60 seconds.
I might end up by grouping them Year, Month, Day and then by time difference is less than 60 seconds .
Like
2020-12-06 17:35:05 5 Liters
2020-12-06 17:43:05 13 Liters
Here is a phpmyaddmin sql dump if it helps
-- phpMyAdmin SQL Dump
-- version 4.6.6deb5
-- https://www.phpmyadmin.net/
--
-- Host: localhost:3306
-- Generation Time: Dec 10, 2020 at 07:27 AM
-- Server version: 10.3.17-MariaDB-0+deb10u1
-- PHP Version: 7.3.11-1~deb10u1
SET SQL_MODE = "NO_AUTO_VALUE_ON_ZERO";
SET time_zone = "+00:00";
/*!40101 SET #OLD_CHARACTER_SET_CLIENT=##CHARACTER_SET_CLIENT */;
/*!40101 SET #OLD_CHARACTER_SET_RESULTS=##CHARACTER_SET_RESULTS */;
/*!40101 SET #OLD_COLLATION_CONNECTION=##COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8mb4 */;
--
-- Database: `mysensors`
--
-- --------------------------------------------------------
--
-- Table structure for table `watermeter`
--
CREATE TABLE `watermeter` (
`id` int(10) NOT NULL,
`liter` int(11) NOT NULL,
`total_liters` int(11) NOT NULL,
`date` date NOT NULL,
`time` time NOT NULL,
`dater` datetime NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
--
-- Dumping data for table `watermeter`
--
INSERT INTO `watermeter` (`id`, `liter`, `total_liters`, `date`, `time`, `dater`) VALUES
(9, 3, 184, '2020-12-06', '16:14:58', '2020-12-06 16:14:58'),
(10, 1, 185, '2020-12-06', '16:15:04', '2020-12-06 16:15:04'),
(11, 3, 188, '2020-12-06', '16:26:49', '2020-12-06 16:26:49'),
(12, 2, 190, '2020-12-06', '16:26:55', '2020-12-06 16:26:55'),
(13, 2, 192, '2020-12-06', '16:27:01', '2020-12-06 16:27:01'),
(14, 1, 193, '2020-12-06', '17:32:16', '2020-12-06 17:32:16'),
(15, 1, 194, '2020-12-06', '17:32:22', '2020-12-06 17:32:22'),
(16, 1, 195, '2020-12-06', '17:32:28', '2020-12-06 17:32:28'),
(17, 1, 196, '2020-12-06', '17:32:35', '2020-12-06 17:32:35'),
(18, 1, 197, '2020-12-06', '17:32:41', '2020-12-06 17:32:41'),
(19, 1, 198, '2020-12-06', '17:32:47', '2020-12-06 17:32:47'),
(20, 1, 199, '2020-12-06', '17:32:53', '2020-12-06 17:32:53'),
(21, 1, 200, '2020-12-06', '17:32:59', '2020-12-06 17:32:59'),
(22, 1, 201, '2020-12-06', '17:35:05', '2020-12-06 17:35:05'),
(23, 1, 202, '2020-12-06', '17:35:17', '2020-12-06 17:35:17'),
(24, 1, 203, '2020-12-06', '17:35:23', '2020-12-06 17:35:23'),
(25, 1, 204, '2020-12-06', '17:35:29', '2020-12-06 17:35:29'),
(26, 1, 205, '2020-12-06', '17:35:41', '2020-12-06 17:35:41'),
(27, 1, 206, '2020-12-06', '17:43:05', '2020-12-06 17:43:05'),
(28, 3, 209, '2020-12-06', '17:43:11', '2020-12-06 17:43:11'),
(29, 2, 211, '2020-12-06', '17:43:17', '2020-12-06 17:43:17'),
(30, 2, 213, '2020-12-06', '17:43:23', '2020-12-06 17:43:23'),
(31, 2, 215, '2020-12-06', '17:43:29', '2020-12-06 17:43:29'),
(32, 3, 218, '2020-12-06', '17:43:36', '2020-12-06 17:43:36'),
(33, 2, 220, '2020-12-06', '17:43:42', '2020-12-06 17:43:42');
--
-- Indexes for dumped tables
--
--
-- Indexes for table `watermeter`
--
ALTER TABLE `watermeter`
ADD PRIMARY KEY (`id`),
ADD KEY `dater` (`dater`);
--
-- AUTO_INCREMENT for dumped tables
--
--
-- AUTO_INCREMENT for table `watermeter`
--
ALTER TABLE `watermeter`
MODIFY `id` int(10) NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=1061;
/*!40101 SET CHARACTER_SET_CLIENT=#OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=#OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=#OLD_COLLATION_CONNECTION */;
UPDATE 1.
Making little progress I think - Totals are not right yet.
SELECT '(a.dater, b.dater)', DATE_FORMAT(a.dater,'%H:%i') AS dater,
YEAR(a.dater),
MONTHNAME(a.dater),
DAY(a.dater),
HOUR(a.dater),
MINUTE(a.dater),
a.time,
SUM(a.liter) as liter
FROM watermeter a
INNER JOIN watermeter b
ON b.dater >= a.dater
WHERE b.dater <= DATE_ADD(a.dater, INTERVAL 60 SECOND)
AND a.date LIKE '2020-12-08' GROUP BY YEAR(a.date), MONTHNAME(a.date), DAY(a.dater), HOUR(a.dater), MINUTE(a.dater)
ORDER BY a.id ASC
Update2
So Update one does not give me the correct result. Tried now the following that I got from : MySQL GROUP BY DateTime +/- 3 seconds but also no joy yet.
SELECT COUNT(liter),DAY(dater),HOUR(dater),MINUTE(dater)
FROM watermeter
JOIN (SELECT watermeter.id, MAX(S.dater) AS ChainStartTime
FROM watermeter
JOIN (SELECT DISTINCT a.dater
FROM watermeter a
LEFT JOIN watermeter b
ON (b.dater >= a.dater - INTERVAL 60 SECOND
AND b.dater < a.dater)
WHERE b.dater IS NULL
AND a.date LIKE '2020-12-06') S
ON watermeter.dater >= S.dater
GROUP BY watermeter.id) GroupingQuery
ON watermeter.id = GroupingQuery.id
GROUP BY GroupingQuery.ChainStartTime

First find the difference in time from the previous row:
select
id,
liter,
total_liters,
dater,
lead(dater) over (order by dater) as "lead",
timediff(dater, lead(dater) over (order by dater)) as d1
from watermeter
order by dater;
output:
+----+-------+--------------+---------------------+---------------------+-----------+
| id | liter | total_liters | dater | lead | d1 |
+----+-------+--------------+---------------------+---------------------+-----------+
| 9 | 3 | 184 | 2020-12-06 16:14:58 | 2020-12-06 16:15:04 | -00:00:06 |
| 10 | 1 | 185 | 2020-12-06 16:15:04 | 2020-12-06 16:26:49 | -00:11:45 |
| 11 | 3 | 188 | 2020-12-06 16:26:49 | 2020-12-06 16:26:55 | -00:00:06 |
| 12 | 2 | 190 | 2020-12-06 16:26:55 | 2020-12-06 16:27:01 | -00:00:06 |
| 13 | 2 | 192 | 2020-12-06 16:27:01 | 2020-12-06 17:32:16 | -01:05:15 |
| 14 | 1 | 193 | 2020-12-06 17:32:16 | 2020-12-06 17:32:22 | -00:00:06 |
| 15 | 1 | 194 | 2020-12-06 17:32:22 | 2020-12-06 17:32:28 | -00:00:06 |
| 16 | 1 | 195 | 2020-12-06 17:32:28 | 2020-12-06 17:32:35 | -00:00:07 |
| 17 | 1 | 196 | 2020-12-06 17:32:35 | 2020-12-06 17:32:41 | -00:00:06 |
etc...
Then determine which times you would like to see, because they have a difference to their previous row which is larger than 60 seconds.
(The id of the column is show in x)
with cte as (
select id,
dater,
liter,
total_liters,
d1,
abs(time_to_sec(d1)) as g1,
case when abs(time_to_sec(d1))>60 then id else 0 end as x
from (
select
id,
liter,
total_liters,
dater,
lead(dater) over (order by dater) as "lead",
timediff(dater, lead(dater) over (order by dater)) as d1
from watermeter
order by dater
) tmp1
)
select * from cte;
output:
+----+---------------------+-------+--------------+-----------+------+----+
| id | dater | liter | total_liters | d1 | g1 | x |
+----+---------------------+-------+--------------+-----------+------+----+
| 9 | 2020-12-06 16:14:58 | 3 | 184 | -00:00:06 | 6 | 0 |
| 10 | 2020-12-06 16:15:04 | 1 | 185 | -00:11:45 | 705 | 10 |
| 11 | 2020-12-06 16:26:49 | 3 | 188 | -00:00:06 | 6 | 0 |
| 12 | 2020-12-06 16:26:55 | 2 | 190 | -00:00:06 | 6 | 0 |
| 13 | 2020-12-06 16:27:01 | 2 | 192 | -01:05:15 | 3915 | 13 |
| 14 | 2020-12-06 17:32:16 | 1 | 193 | -00:00:06 | 6 | 0 |
| 15 | 2020-12-06 17:32:22 | 1 | 194 | -00:00:06 | 6 | 0 |
| 16 | 2020-12-06 17:32:28 | 1 | 195 | -00:00:07 | 7 | 0 |
etc...
Next step is to determine the max(id) which 'belongs' to the x:
with cte as (
select id,
dater,
liter,
total_liters,
d1,
abs(time_to_sec(d1)) as g1,
case when abs(time_to_sec(d1))>60 then id else 0 end as x
from (
select
id,
liter,
total_liters,
dater,
lead(dater) over (order by dater) as "lead",
timediff(dater, lead(dater) over (order by dater)) as d1
from watermeter
order by dater
) tmp1
)
select
id,
dater,
liter,
total_liters
,d1,
g1,
x,
(select min(x)-1 from cte c2 where c2.id>c1.x and c2.x>0) as y
from cte c1
where c1.x<>0
;
output:
+----+---------------------+-------+--------------+-----------+------+----+------+
| id | dater | liter | total_liters | d1 | g1 | x | y |
+----+---------------------+-------+--------------+-----------+------+----+------+
| 10 | 2020-12-06 16:15:04 | 1 | 185 | -00:11:45 | 705 | 10 | 12 |
| 13 | 2020-12-06 16:27:01 | 2 | 192 | -01:05:15 | 3915 | 13 | 20 |
| 21 | 2020-12-06 17:32:59 | 1 | 200 | -00:02:06 | 126 | 21 | 25 |
| 26 | 2020-12-06 17:35:41 | 1 | 205 | -00:07:24 | 444 | 26 | NULL |
+----+---------------------+-------+--------------+-----------+------+----+------+
Note that x and y are the minimum and maximum id for your group.
Finally (this messy stuff):
with cte as (
select id,
dater,
liter,
total_liters,
d1,
abs(time_to_sec(d1)) as g1,
case when abs(time_to_sec(d1))>60 then id else 0 end as x
from (
select
id,
liter,
total_liters,
dater,
lead(dater) over (order by dater) as "lead",
timediff(dater, lead(dater) over (order by dater)) as d1
from watermeter
order by dater
) tmp1
)
select
id,
dater,
(select sum(liter) from watermeter where id between x and y) as rain
from (
select
id,
dater,
liter,
total_liters
,d1,
g1,
x,
(select min(x)-1 from cte c2 where c2.id>c1.x and c2.x>0) as y
from cte c1
where c1.x<>0
) tmp2
;
gives output:
+------+---------------------+------+
| id | dater | rain |
+------+---------------------+------+
| 10 | 2020-12-06 16:15:04 | 6 |
| 13 | 2020-12-06 16:27:01 | 9 |
| 21 | 2020-12-06 17:32:59 | 5 |
| 26 | 2020-12-06 17:35:41 | NULL |
+------+---------------------+------+
I do hope this is close to the expected output...

With the help of Luuk's code and learning a lot about "case" and "lag" and lead" and using nested selects etc. I was able to get a working query for what I wanted.
SET #wgroup := 0;
with cte as (
select
id,
dater,
liter,
total_liters,
d1,
abs(time_to_sec(d1)) as g1,
case when abs(time_to_sec(d1))>60 then #wgroup := #wgroup+1 else #wgroup end as wgroup
from (
select
id,
liter,
total_liters,
dater,
(case
WHEN lag(dater) over (order by dater) IS NULL
THEN timediff(dater, lead(dater) over (order by dater))
ELSE timediff(dater, lag(dater) over (order by dater))
END) AS d1
from watermeter where date like '2020-12-06'
order by dater
) tmp1
)
(select dater,
wgroup,
SUM(liter)
from cte
GROUP BY wgroup)
;
With this I was able to sum all values where the time difference is less than 60 second from without loosing a single line.
+---------------------+--------+------------+
| dater | wgroup | SUM(liter) |
+---------------------+--------+------------+
| 2020-12-06 16:14:58 | 0 | 4 |
| 2020-12-06 16:26:49 | 1 | 7 |
| 2020-12-06 17:32:16 | 2 | 8 |
| 2020-12-06 17:35:05 | 3 | 5 |
| 2020-12-06 17:43:05 | 4 | 308 |
| 2020-12-06 19:19:03 | 5 | 120 |
| 2020-12-06 19:31:29 | 6 | 4 |
| 2020-12-06 19:34:48 | 7 | 1 |
| 2020-12-06 20:30:08 | 8 | 1 |
| 2020-12-06 21:27:06 | 9 | 23 |
+---------------------+--------+------------+

mysql IF value = 0 copy TO

I've got 2 tables: ORDER and ORDERFiltered
ORDER looks like this:
OrderNO. Item Size Quantity
1111111 ITEM22 28 3
2222222 ITEM18 22 4
3333333 ITEM22 26 3
1111111 ITEM22 22 1
1111111 ITEM22 26 2
How can I copy Values to second table (ORDERFiltered) that it looks like this:
OrderNO: Item 22 26 28
1111111 ITEM22 1 2 3
2222222 ITEM18 4 0 0
3333333 ITEM22 0 3 0

Conditional aggregation will do for this (aka pivot)
drop table if exists t;
create table t
(iOrderNO int , Item varchar(10), Size int, Quantity int);
insert into t values
(1111111 , 'ITEM22' , 28, 3),
(2222222 , 'ITEM18' , 22, 4),
(3333333 , 'ITEM22' , 26, 3),
(1111111 , 'ITEM22' , 22, 1),
(1111111 , 'ITEM22' , 26, 2);
select iorderno,item,
max(case when size = 22 then quantity else 0 end) as '22',
max(case when size = 26 then quantity else 0 end) as '26',
max(case when size = 28 then quantity else 0 end) as '28'
from t
group by iorderno,item;
+----------+--------+------+------+------+
| iorderno | item | 22 | 26 | 28 |
+----------+--------+------+------+------+
| 1111111 | ITEM22 | 1 | 2 | 3 |
| 2222222 | ITEM18 | 4 | 0 | 0 |
| 3333333 | ITEM22 | 0 | 3 | 0 |
+----------+--------+------+------+------+
3 rows in set (0.001 sec)

MYSQL Update rows with duplicate value but oldest date

Tried many suggestions to get this to work, difficult to explain so below is the data I have and the result I want to achieve.
I want to update the 'Active' Column to 0 if its not the MAX Ldate.
ID | SNumber | Ldate | Active
4804 188 2015-11-17 1
4806 189 2015-11-25 1
4807 190 2015-11-25 1
4808 191 2015-11-19 1
4809 192 2015-11-19 1
4820 193 2015-11-17 1
4821 193 2016-06-08 1
4830 194 2015-11-17 1
4831 194 2016-06-08 1
4828 195 2015-11-17 1
4829 195 2016-06-08 1
ID SNumber Ldate Active
4804 188 2015-11-17 1
4806 189 2015-11-25 1
4807 190 2015-11-25 1
4808 191 2015-11-19 1
4809 192 2015-11-19 1
4820 193 2015-11-17 0
4821 193 2016-06-08 1
4830 194 2015-11-17 0
4831 194 2016-06-08 1
4828 195 2015-11-17 0
4829 195 2016-06-08 1
I can get all rows with the MAX Ldate by "select ID, SNumber, Ldate from (select * from tbl order by SNumber, Ldate desc) x group by SNumber"
Thanks for taking the time to look!

DROP TABLE IF EXISTS my_table;
CREATE TABLE my_table
(id SERIAL PRIMARY KEY
,SNumber INT NOT NULL
,Ldate DATE NOT NULL
);
INSERT INTO my_table VALUES
(4804,188,'2015-11-17'),
(4806,189,'2015-11-25'),
(4807,190,'2015-11-25'),
(4808,191,'2015-11-19'),
(4809,192,'2015-11-19'),
(4820,193,'2015-11-17'),
(4821,193,'2016-06-08'),
(4830,194,'2015-11-17'),
(4831,194,'2016-06-08'),
(4828,195,'2015-11-17'),
(4829,195,'2016-06-08');
SELECT x.*
, COALESCE(x.ldate = y.ldate,0) active
FROM my_table x
LEFT
JOIN
( SELECT snumber
, MAX(ldate) ldate
FROM my_table
GROUP
BY snumber
) y
ON y.snumber = x.snumber
AND y.ldate = x.ldate;
+------+---------+------------+--------+
| id | SNumber | Ldate | active |
+------+---------+------------+--------+
| 4804 | 188 | 2015-11-17 | 1 |
| 4806 | 189 | 2015-11-25 | 1 |
| 4807 | 190 | 2015-11-25 | 1 |
| 4808 | 191 | 2015-11-19 | 1 |
| 4809 | 192 | 2015-11-19 | 1 |
| 4820 | 193 | 2015-11-17 | 0 |
| 4821 | 193 | 2016-06-08 | 1 |
| 4830 | 194 | 2015-11-17 | 0 |
| 4831 | 194 | 2016-06-08 | 1 |
| 4828 | 195 | 2015-11-17 | 0 |
| 4829 | 195 | 2016-06-08 | 1 |
+------+---------+------------+--------+
I can't think why you'd store this, but it's easy enough to change the above to an UPDATE. It might look something like this (obviously, you'd need to alter the table design above first)...
UPDATE my_table x
LEFT
JOIN
( SELECT snumber
, MAX(ldate) ldate
FROM my_table
GROUP
BY snumber
) y
ON y.snumber = x.snumber
AND y.ldate = x.ldate
SET x.active = 0
WHERE y.snumber IS NULL;
But I think I would normally use an INNER JOIN for an UPDATE, in which case it might look like this (perhaps bound up inside a transaction)...
UPDATE my_table SET active = 0;
UPDATE my_table x
JOIN
( SELECT snumber
, MAX(ldate) ldate
FROM my_table
GROUP
BY snumber
) y
ON y.snumber = x.snumber
AND y.ldate = x.ldate
SET x.active = 1;

Query to display summary of each user per month from January to December [duplicate]

This question already has an answer here:
MySQL pivot row into dynamic number of columns
(1 answer)
Closed 6 years ago.
Am trying to write a query to display all the vehicles fuel consumption per month from January to December depending with the year picked by a user. the objective is to get something like this.
+-------------+---------------------------------------------------------+
| Vehicle | Jan | Feb | Mar | Arp | May | Jun | Jul | Aug | Sep |...|
+-------------+---------------------------------------------------------+
| NY001 | 30 | 44 | 54 | 98 | 12 | 37 | 76 | | 32 |...|
| NY002 | 30 | 65 | 99 | | 12 | 85 | | 13 | 52 |...|
| NY003 | 30 | 95 | 24 | 98 | | 55 | 76 | 43 | |...|
| NY004 | | 22 | 97 | | 12 | 28 | | | 12 |...|
| NY005 | 30 | 36 | 15 | 98 | 12 | 65 | 36 | 23 | 66 |...|
| NY006 | | 76 | 34 | 98 | 12 | 75 | 16 | 93 | 97 |...|
+-------------+---------------------------------------------------------+
My table syntax is like below
CREATE TABLE `COSTING` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`Vehicle` varchar(100) NOT NULL DEFAULT '',
`Fueldate` datetime NOT NULL,
`Costcenter` varchar(200) NOT NULL DEFAULT '',
`Code` varchar(100) NOT NULL DEFAULT '',
`Cost` int(100) NOT NULL DEFAULT '0',
`FuelLitters` varchar(100) NOT NULL DEFAULT '',
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=24 DEFAULT CHARSET=latin1;
How do i do this using mysql query. Please help me. am new in this.
Sample data:
Vehicle | Fueldate | Cost | FuelLitters
----------------------------------------
s213 2016-11-10 17536 96.87
S055 2016-11-23 350 12.66
s213 2016-11-15 7204 33.87
s213 2016-11-24 10827 34.66
S530 2016-11-29 4063 42.81
s515 2016-11-26 20599 250
S513 2016-11-26 9108 110.94
S213 2016-12-29 4400 46.36
S212 2016-11-29 6708 70.68
S211 2016-12-29 3059 32.23
S210 2016-11-28 3950 41.62
S513 2016-12-29 2428 25.58
s160 2016-11-25 4002 42.17
MGT02T 2016-11-26 20804 219.21

Edit
After shadow's comment here another solution without subquery :
select c.Vehicle,
sum(case when DATE_FORMAT(Fueldate, "%c") = 1 then FuelLitters end) as JAN,
sum(case when DATE_FORMAT(Fueldate, "%c") = 2 then FuelLitters end) as FEB,
sum(case when DATE_FORMAT(Fueldate, "%c") = 3 then FuelLitters end) as MAR,
sum(case when DATE_FORMAT(Fueldate, "%c") = 4 then FuelLitters end) as APR,
sum(case when DATE_FORMAT(Fueldate, "%c") = 5 then FuelLitters end) as MAY,
sum(case when DATE_FORMAT(Fueldate, "%c") = 6 then FuelLitters end) as JUN,
sum(case when DATE_FORMAT(Fueldate, "%c") = 7 then FuelLitters end) as JUL,
sum(case when DATE_FORMAT(Fueldate, "%c") = 8 then FuelLitters end) as AUG,
sum(case when DATE_FORMAT(Fueldate, "%c") = 9 then FuelLitters end) as SEP,
sum(case when DATE_FORMAT(Fueldate, "%c") = 10 then FuelLitters end) as OCT,
sum(case when DATE_FORMAT(Fueldate, "%c") = 11 then FuelLitters end) as NOV,
sum(case when DATE_FORMAT(Fueldate, "%c") = 12 then FuelLitters end) as `DEC`
from COSTING c
group by Vehicle
For explanations : you have not the columns in your table, so you have to calculate it on the fly to get result. For each month we create a case when selection to get only the data from the month. Grouping it by vehicle at the end allow us to make the sum by vehicle.

Still show the proper set of time even if there's no entry for that time

I have this query where it gets the average and group the values by 15 mins from 12 AM to 11:45 PM.
SELECT FROM_UNIXTIME(t_stamp/1000, '%m/%d/%Y %l:%i %p') as t_stamp,
ROUND(AVG(CASE WHEN id = '001' THEN value END),2) Value1,
ROUND(AVG(CASE WHEN id = '002' THEN value END),2) Value2,
ROUND(AVG(CASE WHEN id = '003' THEN value END),2) Value3
FROM table1
WHERE tagid IN ("001", "002", "003") and
date(from_unixtime(t_stamp/1000)) BETWEEN "2014-05-01" AND "2014-05-01"
GROUP BY DATE(from_unixtime(t_stamp/1000)), HOUR(from_unixtime(t_stamp/1000)), MINUTE(from_unixtime(t_stamp/1000)) DIV 15
The output looks like this
t_stamp | Value1 | Value2 | Value3
05/01/2014 12:00 AM | 199 | 99 | 100
05/01/2014 12:15 AM | 299 | 19 | 140
05/01/2014 12:30 AM | 399 | 59 | 106
05/01/2014 12:45 AM | 499 | 59 | 112
.
.
.
05/01/2014 11:00 PM | 149 | 199 | 100
05/01/2014 11:15 PM | 599 | 93 | 123
05/01/2014 11:30 PM | 129 | 56 | 150
05/01/2014 11:45 PM | 109 | 60 | 134
It works fine but I've noticed that sometimes if there's no entry for like the time 12:30 instead of showing
t_stamp | Value1 | Value2 | Value3
05/01/2014 12:00 AM | 199 | 99 | 100
05/01/2014 12:15 AM | 299 | 19 | 140
05/01/2014 12:30 AM | Null | Null | Null
05/01/2014 12:45 AM | 499 | 59 | 112
It will show the set of time like this:
t_stamp | Value1 | Value2 | Value3
05/01/2014 12:00 AM | 199 | 99 | 100
05/01/2014 12:15 AM | 299 | 19 | 140
05/01/2014 12:33 AM | 122 | 141 | 234
05/01/2014 12:45 AM | 499 | 59 | 112
What I would like to happen is when there's no time for that 15 min group it will still show the proper set of time and then just show null on the column values. The output I would like is like this:
t_stamp | Value1 | Value2 | Value3
05/01/2014 12:00 AM | 199 | 99 | 100
05/01/2014 12:15 AM | 299 | 19 | 140
05/01/2014 12:30 AM | Null | Null | Null
05/01/2014 12:45 AM | 499 | 59 | 112
How can I do this?
Thank You.

You need a table that's a source of cardinal numbers as a start for this. For the moment let's assume it exists, and it's called cardinal.
Then, you need to create a query (a virtual table) that will return rows with timestamps every fifteen minutes, starting with the earliest relevant timestamp and ending with the latest. Here's how to do that for your query.
SELECT '2014-05-01' + INTERVAL (cardinal.n * 15) MINUTE as t_stamp
FROM cardinal
WHERE cardinal.n <= 24*4
Then you need to JOIN that virtual table to your existing query, as follows
SELECT DATE_FORMAT(t_stamp.t_stamp, '%m/%d/%Y %l:%i %p') t_stamp,
ROUND(AVG(CASE WHEN id = '001' THEN value END),2) Value1,
ROUND(AVG(CASE WHEN id = '002' THEN value END),2) Value2,
ROUND(AVG(CASE WHEN id = '003' THEN value END),2) Value3
FROM table1 AS t
LEFT JOIN (
SELECT '2014-05-01' + INTERVAL (cardinal.n * 15) MINUTE as t_stamp
FROM cardinal
WHERE cardinal.n <= 24*4
) AS t_stamp
ON t_stamp.t_stamp = FROM_UNIXTIME(t.t_stamp/1000)
WHERE tagid IN ("001", "002", "003")
AND date(from_unixtime(t_stamp/1000)) BETWEEN "2014-05-01" AND "2014-05-01"
GROUP BY DATE(from_unixtime(t_stamp/1000)),
HOUR(from_unixtime(t_stamp/1000)),
MINUTE(from_unixtime(t_stamp/1000)) DIV 15
Notice that the LEFT JOIN makes sure the rows will NULL values from your original query get included in the result set.
Now, where does this magical cardinal table come from?
You can generate it as two views, like this. This particular view generates numbers from 0 to 100 000, which is more than enough for quarters of hours for a year.
CREATE OR REPLACE VIEW cardinal10 AS
SELECT 0 AS N UNION
SELECT 1 AS N UNION
SELECT 2 AS N UNION
SELECT 3 AS N UNION
SELECT 4 AS N UNION
SELECT 5 AS N UNION
SELECT 6 AS N UNION
SELECT 7 AS N UNION
SELECT 8 AS N UNION
SELECT 9 AS N;
CREATE OR REPLACE VIEW cardinal AS
SELECT A.N + 10*(B.N + 10*(C.N + 10*(D.N + 10*(E.N)))) AS N
FROM cardinal10 A,cardinal10 B,cardinal10 C,
cardinal10 D,cardinal10 E;
Here's a writeup on the topic.
http://www.plumislandmedia.net/mysql/filling-missing-data-sequences-cardinal-integers/