I would like to get the count of a column based on its recent status.
Please see table structure below:
id | visible | date
1 | 1 | 2021-07-22
2 | 1 | 2021-07-23
3 | 0 | 2021-07-24
4 | 1 | 2021-07-25
5 | 0 | 2021-07-26
6 | 0 | 2021-07-27
For example, if I query
SELECT COUNT(visible) AS latest_not_visible WHERE date = '2021-07-26'
then it should return
latest_not_visible
1
Since it only counts that date as not visible, it disregarded the count on 07/24 since 07/25 is visible
But if I query
SELECT COUNT(visible) AS latest_not_visible WHERE date = '2021-07-27'
latest_not_visible
2
since 07/26 and 07/27 are both non-visible and no date in between is visible
I already had the solution to the problem, but I would need help in optimizing this function:
IIF(datediff
(day,
(SELECT MAX(date) FROM t1 WHERE (visible = 0 OR visible = '-1' OR visible = '-3') AND item_id = vp.item_id AND [date] <= vp.date),
(SELECT MAX(date) FROM t1 WHERE visible = 1 AND item_id = vp.item_id AND [date] <= vp.date)) IS NULL
OR datediff(day,
(SELECT MAX(date) FROM t1 WHERE (visible = 0 OR visible = '-1' OR visible = '-3') AND item_id = vp.item_id AND [date] <= vp.date),
(SELECT MAX(date) FROM t1 WHERE visible = 1 AND item_id = vp.item_id AND [date] <= vp.date)) < 0,
(SELECT COUNT(1) FROM t1 WHERE (visible = 0 OR visible = '-1' OR visible = '-3') AND item_id = vp.item_id AND [date] <= vp.date), 0)
AS times_not_visible,
table vp is the original table same with t1
Rather than counting them, you're better to calculate the number of days between the last visible and not visible on or before that date. So something like this...
SELECT DATEDIFF(
SELECT MAX(date) FROM YourTable WHERE visible = 0 AND date <= '2021-07-27'),
SELECT MAX(date) FROM YourTable WHERE visible = 1 AND date <= '2021-07-27')
) as latest_not_visible;
Find the latest visible date earlier than the given date and count all rows in between those two dates:
SET #dt = '2021-07-25';
SELECT COUNT(*)
FROM t
WHERE date <= (SELECT date FROM t WHERE date = #dt AND visible = 0)
AND date > (SELECT date FROM t WHERE date < #dt AND visible = 1 ORDER BY date DESC LIMIT 1)
SQL Fiddle
A solution with GROUP_CONCAT():
SELECT CHAR_LENGTH(visible) - CHAR_LENGTH(TRIM(TRAILING '0' FROM visible)) latest_not_visible
FROM (
SELECT GROUP_CONCAT(visible ORDER BY date SEPARATOR '') visible
FROM tablename
WHERE date <= ?
) t
Change ? to the date you want.
See the demo.
You can use window functions:
select count(*)
from (select t.*,
max(case when visible = 1 then date end) over (order by date) as max_visible_date
from t
where date <= '2021-07-26' -- or whatever
) t
where visible = 0 and
(date > max_visible_date or max_visible_date is null);
Note that this version only mentions the date once. It also works for if there are no rows with visible = 1, and it works on any number of rows.
Related
I have a cursor pagination implemented with MySQL (version 8.0), which works fine as long as there are no null values involved.
Here's my example data (id is a random UUID, date is date, time is time):
id | date | time
--------------------------
68 | 2017-10-28 | 22:00:00
d3 | 2017-11-03 | null
dd | 2017-11-03 | 21:45:00
62 | 2017-11-04 | 14:00:00
a1 | 2017-11-04 | 19:40:00
The cursor I use always consists of all three columns.
I use this query to get the next results (after the cursor):
SELECT * FROM table
WHERE (date > cursor.date)
OR (date = cursor.date AND time > cursor.time)
OR (date = cursor.date AND time = cursor.time AND id > cursor.id)
ORDER BY date ASC, time ASC, id ASC
And this query for prev results (before the cursor):
SELECT * FROM table
WHERE (date < cursor.date)
OR (date = cursor.date AND time < cursor.time)
OR (date = cursor.date AND time = cursor.time AND id < cursor.id)
ORDER BY date DESC, time DESC, id DESC
When using the prev query with cursor [id = dd, date = 2017-11-03, time = 21:45:00] it won't return the row with id = d3, because time is null, and this won't get selected by time < cursor.time.
Though I tried using time < cursor.time OR time IS NULL instead of time < cursor.time to include rows with null values. Which seems to fix this particular problem, but then creates a new problem: When using the prev query with cursor [id = d3, date = 2017-11-03, time = null], because now the result contains the row of the provided cursor.
I hope there's an easy solution for this. There seems to be no examples or tutorials on the web that deal with null values in cursor pagination.
Note: For the solution it doesn't matter if null will be sorted before or after non-null values, as long as it's consistent. (MySQL's default ordering is null < non-null)
I am not going to touch the topic of using cursors for pagination. There are alternatives, such as limit/offset.
But my recommendation for your queries is to use coalesce(), assigning a fake time for the comparison. MySQL makes this somewhat simple, because it supports time values in excess of 24 hours. And those would not be valid values for a date/time combination.
So:
SELECT *
FROM table
WHERE (date > cursor.date) OR
(date = cursor.date AND COALESCE(time, '24:00:00') > COALESCE(cursor.time, '24:00:00')) OR
(date = cursor.date AND COALESCE(time, '24:00:00') = COALESCE(cursor.time, '24:00:00') AND id > cursor.id)
ORDER BY date ASC, time ASC, id ASC
A more concise WHERE clause would be:
WHERE (date, COALESCE(time, '24:00:00'), id) > (cursor.date, COALESCE(cursor.time, '24:00:00'), cursor.id)
I'm a bit late to the party but give the following a try. The cursor logic needs to be adjusted for nullable columns and when a cursor row returns a null value for the time column.
Both examples listed below:
declare #cursorID nvarchar(2)
declare #cursorDate date
declare #cursorTime time(0)
declare #table table(id nvarchar(2), date date, time time(0))
insert into #table
values
('68', '2017-10-28', '22:00:00'),
('d3', '2017-11-03', NULL),
('dd', '2017-11-03', '21:45:00'),
('62', '2017-11-04', '14:00:00'),
('a1', '2017-11-04', '19:40:00')
--IF SELECTING A ROW WITH A NON-NULL VALUE FOR TIME
set #cursorID = 'dd'
set #cursorDate = (select date from #table where id = #cursorID)
set #cursorTime = (select time from #table where id = #cursorID)
--ASCENDING CURSOR VALUES
select * from #table
where
date >= #cursorDate
and (date > #cursorDate or (time >= #cursorTime
and (time > #cursorTime or id > #cursorID)))
order by
date asc, time asc, id asc
--DESCENDING CURSOR VALUES
select * from #table
where
date <= #cursorDate
and (date < #cursorDate or ((time <= #cursorTime or time is null)
and ((time < #cursorTime or time is null) or id < #cursorID)))
order by
date desc, time desc, id desc
--IF SELECTING A ROW WITH A NON VALUE FOR TIME
set #cursorID = 'd3'
set #cursorDate = (select date from #table where id = #cursorID)
set #cursorTime = (select time from #table where id = #cursorID)
--ASCENDING CURSOR VALUES
select * from #table
where
date >= #cursorDate
and (date > #cursorDate or (time is not null
or (time is null and id > #cursorID)))
order by
date asc, time asc, id asc
--DESCENDING CURSOR VALUES
select * from #table
where
date <= #cursorDate
and (date < #cursorDate or (time is null
and (id < #cursorID)))
order by
date desc, time desc, id desc
Add another column to the table. Make it a DATETIME. Combine date and time into it when not NULL; combine date with some particular time when NULL. Then your cursor has two columns to work with and no nulls.
If you have a reasonably recent version of MySQL, you can use a "generated stored" column, thereby avoiding any code changes.
And be sure to have INDEX(datetime, id).
If you are using MySQL 8.0 then you can consider to use row_number() window funciton create an unique sequential id (rn) for each row. Then just pass the rn for current row to get the previous rows.
Schema and insert statements:
create table cursortable( id varchar(10), date date, time time);
insert into cursortable values('68' , '2017-10-28' , '22:00:00');
insert into cursortable values('d3' , '2017-11-03' , null);
insert into cursortable values('dd' , '2017-11-03' , '21:45:00');
insert into cursortable values('62' , '2017-11-04' , '14:00:00');
insert into cursortable values('a1' , '2017-11-04' , '19:40:00');
Query to get the result for fist time:
select *,row_number()over(order by date,time,id)rn from cursortable
Output:
id
date
time
rn
68
2017-10-28
22:00:00
1
d3
2017-11-03
null
2
dd
2017-11-03
21:45:00
3
62
2017-11-04
14:00:00
4
a1
2017-11-04
19:40:00
5
Query to get the previous rows for cursor [id = dd, date = 2017-11-03, time = 21:45:00, rn=3] with only cursor [rn=3]:
with cte as
(
select *,row_number()over(order by date,time,id)rn from cursortable
)
select * from cte where rn<3
Output:
id
date
time
rn
68
2017-10-28
22:00:00
1
d3
2017-11-03
null
2
db<>fiddle here
If you don't want to introduce an computed column into your code then please try the below solution considering all three columns cursor [id = dd, date = 2017-11-03, time = 21:45:00]
Query:
with cte as
(
select *,row_number()over(order by date,time,id)rn from cursortable
)
,cte2 as
(
select * from cte where id='dd' and date= '2017-11-03' and time= '21:45:00'
)
select cte.id,cte.date,cte.time from cte inner join cte2 on cte.rn<cte2.rn
Output:
id
date
time
68
2017-10-28
22:00:00
d3
2017-11-03
null
db<>fiddle here
Your code would be like below:
with cte as
(
select *,row_number()over(order by date,time,id)rn from cursortable
)
,cte2 as
(
select * from cte where id=cursor.id and date= cursor.date and time= cursor.time
)
select cte.id,cte.date,cte.time from cte inner join cte2 on cte.rn<cte2.rn
My database has two columns ID and Timestamp.
4 1597228600
8 1597228700
12 1597228700
11 1597228800
14 1597228800
9 1597228900
10 1597228900
1 1597228900
2 1597229000
I need to get next (previous) record of the given id and timestamp ordered by timestamp. If the timestamp has duplicates, the record with higher(lower) id should be returned.
In the example Next and Prev records of the 11(1597228800) are 14 and 12. Next and Prev records of the 14(1597228800) are 1 and 11.
I tried to use CASE condition with subquery, but this solution has issues
SELECT id
FROM tbl
WHERE timestamp_value >= '1597228800'
AND id > (case when ( SELECT min(id) min_id FROM tbl WHERE id > 11 AND timestamp_value = 1597228800) is null then 0 else 11 end)
ORDER BY timestamp_value
LIMIT 1
I think that this will do:
select t.*
from tablename t
cross join (select * from tablename where id = ?) i
where t.id in (
(
select id from tablename
where (id < i.id and timestamp = i.timestamp) or timestamp < i.timestamp
order by timestamp desc, id desc limit 1
),
(
select id from tablename
where (id > i.id and timestamp = i.timestamp) or timestamp > i.timestamp
order by timestamp, id limit 1
)
)
Replace ? with the id that you want to search for.
The 2 subqueries return the ids of the previous and the next id of ?.
See the demo.
Below I cannot do Group by Date the following figures.
I have tried to put Group By in different lines, but not working.
SELECT SUM(a.NetAmount) AS TotalDonation
FROM (
SELECT
(
CASE WHEN bt.BalanceTransactionCurrencyID = 17
THEN bt.BalanceTransactionNet
ELSE
bt.BalanceTransactionNet * (SELECT TOP 1 ExrateValue FROM Exrate WHERE ExrateDate < bt.BalanceTransactionCreated AND bt.BalanceTransactionCurrencyID = CurrencyID ORDER BY ExrateDate Desc)
END
) AS NetAmount
FROM Charge as ch
JOIN BalanceTransaction as bt ON (ch.BalanceTransactionID = bt.BalanceTransactionID)
WHERE ch.ChargeCreatedDate BETWEEN '3-1-2019' AND '3-31-2019'
) AS a
I wanted to see:
Days Total Amount
March 1 xxxx
March 2 xxxx
March 3 xxx
MySQL does not use TOP. Use LIMIT:
SELECT ChargeCreatedDate, SUM(netamount)
FROM (SELECT ch.ChargeCreatedDate,
(CASE WHEN bt.BalanceTransactionCurrencyID = 17
THEN bt.BalanceTransactionNet
ELSE bt.BalanceTransactionNet * (SELECT e.ExrateValue
FROM Exrate e
WHERE e.ExrateDate < bt.BalanceTransactionCreated AND
e.CurrencyID = bt.BalanceTransactionCurrencyID
ORDER BY ExrateDate Desc
LIMIT 1
)
END) AS NetAmount
FROM Charge ch JOIN
BalanceTransaction bt
ON ch.BalanceTransactionID = bt.BalanceTransactionID
WHERE ch.ChargeCreatedDate BETWEEN '2019-03-01' AND '2019-03-31'
) chtbt
GROUP BY ChargeCreatedDate;
If you happen to be using SQL Server, you can replace the LIMIT 1 with FETCH FIRST 1 ROW ONLY.
The following code will hopefully display what you are looking for
SELECT a.Days AS Days, SUM(cast(a.NetAmount as decimal(16,9))) AS TotalDonation
FROM (
SELECT
(
CASE WHEN bt.BalanceTransactionCurrencyID = 17
THEN bt.BalanceTransactionNet
ELSE
bt.BalanceTransactionNet * (SELECT TOP 1 ExrateValue FROM Exrate WHERE ExrateDate < bt.BalanceTransactionCreated AND bt.BalanceTransactionCurrencyID = CurrencyID ORDER BY ExrateDate Desc)
END
) AS NetAmount,
ch.ChargeCreatedDate as Days
FROM Charge as ch
JOIN BalanceTransaction as bt ON (ch.BalanceTransactionID = bt.BalanceTransactionID)
WHERE ch.ChargeCreatedDate BETWEEN '3-1-2019' AND '3-31-2019'
) AS a GROUP BY a.Days
This should be sufficient. You need to SELECT the desired value in the query in order to get it to show up. Also when using the SUM() function you need to specify what the group value will be.
How to do running total or cumulative sum from this query? Is it possible to run by correlated subquery? The cumulative sum results will be show as 'New value'.
SELECT
sum(data2.quantity/1000) AS UnitMT
FROM
data2
INNER JOIN itmnocate ON data2.item = itmnocate.ItemNumber
and Source in ('imported','local','by product')
WHERE date1 >= DATE_FORMAT('2018-04-12', '%Y-01-01')- INTERVAL 1 YEAR
AND date1 <= DATE_FORMAT('2018-04-12', '%Y-12-31') - INTERVAL 1 YEAR
AND data2.unit = 'KG'
and data2.customeracc not in (select Customeraccount from custlist WHERE Custcat = 'bcsb')
GROUP BY month(date1)
This is a cumulative problem If your mysql version support window function you can use SUM with window function to do cumulative.
The DATE1 column can be used as the basis for order by to do cumulative.
SELECT *,
sum(UnitMT) over (order by month(date1)) 'New value'
FROM T t1
sqlfiddle
If your mysql version didn't support window function, you can try to use subquery in select to do cumulative.
CREATE TABLE T( date1 DATE,UnitMT int);
INSERT INTO T VALUES ('2017-01-01',66535);
INSERT INTO T VALUES ('2017-02-01',108337);
INSERT INTO T VALUES ('2017-03-01',132767);
INSERT INTO T VALUES ('2017-04-01',100687);
INSERT INTO T VALUES ('2017-05-01',125151);
Query 1:
SELECT *,
(SELECT SUM(UnitMT) FROM T tt WHERE month(tt.date1) <= month(t1.date1)) 'New value'
FROM T t1
Results:
| date1 | UnitMT | New value |
|------------|--------|-----------|
| 2017-01-01 | 66535 | 66535 |
| 2017-02-01 | 108337 | 174872 |
| 2017-03-01 | 132767 | 307639 |
| 2017-04-01 | 100687 | 408326 |
| 2017-05-01 | 125151 | 533477 |
Note
T symbol your current result set data.
You could try this query, it's efficient and works an all MySQL versions:
select #cumSum := 0;
select UnitMT, #cumSum := #cumSum + UnitMT
from tbl
order by date1;
Demo
For your specific problem, you can use variables and a subquery:
SELECT mon, UnitMT,
(#sum := #sum + UnitMT) as running_sum
FROM (SELECT month(date1) as mon, sum(data2.quantity/1000) AS UnitMT
FROM data2 INNER JOIN
itmnocate
ON data2.item = itmnocate.ItemNumber AND
Source IN ('imported', 'local', 'by product')
WHERE date1 >= DATE_FORMAT('2018-04-12', '%Y-01-01') - INTERVAL 1 YEAR AND
date1 <= DATE_FORMAT('2018-04-12', '%Y-12-31') - INTERVAL 1 YEAR AND
data2.unit = 'KG' AND
data2.customeracc not in (select Customeraccount from custlist WHERE Custcat = 'bcsb')
GROUP BY month(date1)
ORDER BY month(date1)
) m CROSS JOIN
(SELECT #sum := 0) params;
I have a table with minute-by-minute data from an IOT device. Every minute there is a new row with a timestamp and a value that represents a metric. The metric starts at 0 and increments for a while before it resets and starts over.
When I plot it, it looks like the picture. I want to find the local maximum value of each run, as the blue circles indicate.
Is it possible to find and group the consecutive rows where the metric is > 0 and then find the maximum of each group?
Update
Table structure:
+-------------+------------------+
| Field | Type |
+-------------+------------------+
| id | int(10) unsigned |
| timestamp | timestamp |
| metric_name | varchar(32) |
| value | int(10) |
+-------------+------------------+
This is based on the following assumptions:
Id is a perfectly sequential integer (with no gaps)
You want to get the value logged directly before the 0 value
Code:
SELECT *
FROM metrics m1
WHERE m.id IN (
SELECT m2.id - 1
FROM metrics m2
WHERE m1.value = 0)
I join everything that isnt zero before a timestamp where it is zero, then I find the ones with no values inbetween that 0 and the last one..
SELECT
value,
timestamp
FROM
metrics
LEFT JOIN metrics zeros
on metrics.time < zeros.time
and zeros.value = 0
LEFT JOIN metrics betweenZero
on metrics.time < betweenZero.time
and betweenZero.time < zeros.time
INNER JOIN metrics noBetweens
on table.id = noBetweens.id
and betweenZero.id IS NULL
If you need it for a paritulcar metric_name, WHERE metric_name = the_metric_nameon the end.
This should give you the max value per group along with start time and end time of each window with only 1 pass over the data.
select metric_name, max(value) value, max(start_group) start_time, max(end_group) end_time from(
select metric_name, value,
case when #prev_ts is not null then #prev_ts end prev_ts,
case when value = 0 then #ts := timestamp end as start_group,
#ts as grouping,
#prev_ts := timestamp end_group
from metric join (select #prev_ts := null as p) prev
order by timestamp
) q
group by metric_name, grouping;
This will create a sample data set of 1000 rows, that resets every minute.
insert into metric(timestamp, metric_name, value)
select now() - interval rn second, 'pressure', v
from(
select #rn := #rn + 1 rn, mod(1000 - #rn,60) * pow(1000 - mod(#rn,121),1) v
from table_with_at_least_1000_rows
join (select #rn := 0) rn
limit 1000
) q
;
Try this:
SELECT
T.min_id
,T.max_id
,MAX(M.value) as local_max
FROM
metrics M
JOIN (
SELECT
id as min_id
,(
SELECT MIN(id) FROM Metrics MI
WHERE
MI.id > MO.id
AND MI.value = 0) as max_id
FROM Metrics MO
WHERE
value = 0
)T ON M.id BETWEEN T.min_id AND T.max_id
GROUP BY
T.min_id, T.max_id
My solution doesn't care about gaps but I am assuming that the sequence of ids is monotonic, that is they increase along the series by time. (You could probably substitute id for timestamp in the query even.) I had made a few minor syntax-type errors that I have since corrected since my first attempt and I have tested it with a simple Fiddle. I think it works.
select t0.*
from
T t0 inner join
(
select max_z, max(id) as max_id, max(value) as local_max
from
(
select
id, value,
(
select max(t2.id) as max_id from T t2
where t2.id < t.id and t2.value = 0
) as max_z
from T t
where t.value <> 0
) p /* partitions */
group by p.max_z
) x /* extrema */
on t0.id between max_z and max_id and t0.value = x.local_max
Btw it returns all the rows when there's a tie for the local maximum.
http://sqlfiddle.com/#!9/de832/2