Finding local maximum between zero-values using SQL - mysql

I have a table with minute-by-minute data from an IOT device. Every minute there is a new row with a timestamp and a value that represents a metric. The metric starts at 0 and increments for a while before it resets and starts over.
When I plot it, it looks like the picture. I want to find the local maximum value of each run, as the blue circles indicate.
Is it possible to find and group the consecutive rows where the metric is > 0 and then find the maximum of each group?
Update
Table structure:
+-------------+------------------+
| Field | Type |
+-------------+------------------+
| id | int(10) unsigned |
| timestamp | timestamp |
| metric_name | varchar(32) |
| value | int(10) |
+-------------+------------------+

This is based on the following assumptions:
Id is a perfectly sequential integer (with no gaps)
You want to get the value logged directly before the 0 value
Code:
SELECT *
FROM metrics m1
WHERE m.id IN (
SELECT m2.id - 1
FROM metrics m2
WHERE m1.value = 0)

I join everything that isnt zero before a timestamp where it is zero, then I find the ones with no values inbetween that 0 and the last one..
SELECT
value,
timestamp
FROM
metrics
LEFT JOIN metrics zeros
on metrics.time < zeros.time
and zeros.value = 0
LEFT JOIN metrics betweenZero
on metrics.time < betweenZero.time
and betweenZero.time < zeros.time
INNER JOIN metrics noBetweens
on table.id = noBetweens.id
and betweenZero.id IS NULL
If you need it for a paritulcar metric_name, WHERE metric_name = the_metric_nameon the end.

This should give you the max value per group along with start time and end time of each window with only 1 pass over the data.
select metric_name, max(value) value, max(start_group) start_time, max(end_group) end_time from(
select metric_name, value,
case when #prev_ts is not null then #prev_ts end prev_ts,
case when value = 0 then #ts := timestamp end as start_group,
#ts as grouping,
#prev_ts := timestamp end_group
from metric join (select #prev_ts := null as p) prev
order by timestamp
) q
group by metric_name, grouping;
This will create a sample data set of 1000 rows, that resets every minute.
insert into metric(timestamp, metric_name, value)
select now() - interval rn second, 'pressure', v
from(
select #rn := #rn + 1 rn, mod(1000 - #rn,60) * pow(1000 - mod(#rn,121),1) v
from table_with_at_least_1000_rows
join (select #rn := 0) rn
limit 1000
) q
;

Try this:
SELECT
T.min_id
,T.max_id
,MAX(M.value) as local_max
FROM
metrics M
JOIN (
SELECT
id as min_id
,(
SELECT MIN(id) FROM Metrics MI
WHERE
MI.id > MO.id
AND MI.value = 0) as max_id
FROM Metrics MO
WHERE
value = 0
)T ON M.id BETWEEN T.min_id AND T.max_id
GROUP BY
T.min_id, T.max_id

My solution doesn't care about gaps but I am assuming that the sequence of ids is monotonic, that is they increase along the series by time. (You could probably substitute id for timestamp in the query even.) I had made a few minor syntax-type errors that I have since corrected since my first attempt and I have tested it with a simple Fiddle. I think it works.
select t0.*
from
T t0 inner join
(
select max_z, max(id) as max_id, max(value) as local_max
from
(
select
id, value,
(
select max(t2.id) as max_id from T t2
where t2.id < t.id and t2.value = 0
) as max_z
from T t
where t.value <> 0
) p /* partitions */
group by p.max_z
) x /* extrema */
on t0.id between max_z and max_id and t0.value = x.local_max
Btw it returns all the rows when there's a tie for the local maximum.
http://sqlfiddle.com/#!9/de832/2

Related

get the range of sequence values in table column

I have a list of value in my column. And want to query the range.
Eg. If values are 1,2,3,4,5,9,11,12,13,14,17,18,19
I want to display
1-5,9,11-14,17-19
Assuming that each value is stored on a separate row, you can use some gaps-and-island technique here:
select case when min(val) <> max(val)
then concat(min(val), '-', max(val))
else min(val)
end val_range
from (select val, row_number() over(order by val) rn from mytable) t
group by val - rn
order by min(val)
The idea is to build groups of consecutive values by taking the difference between the value and an incrementing rank, which is computed using row_number() (available in MySQL 8.0):
Demo on DB Fiddle:
| val_range |
| :-------- |
| 1-5 |
| 9 |
| 11-14 |
| 17-19 |
In earlier versions, you can emulate row_number() with a correlated subquery, or a user variable. The second option goes like:
select case when min(val) <> max(val)
then concat(min(val), '-', max(val))
else min(val)
end val_range
from (select #rn := 0) x
cross join (
select val, #rn := #rn + 1 rn
from (select val from mytable order by val) t
) t
group by val - rn
order by min(val)
As a complement to other answers:
select dn.val as dnval, min(up.val) as upval
from mytable up
join mytable dn
on dn.val <= up.val
where not exists (select 1 from mytable a where a.val = up.val + 1)
and not exists (select 1 from mytable b where b.val = dn.val - 1)
group by dn.val
order by dn.val;
1 5
9 9
11 14
17 19
Needless to say, but using an OLAP function like #GNB does, is orders of magnitude more efficient.
A short article on how to mimic OLAP functions in MySQL < 8 can be found at:
mysql-row_number
Fiddle
EDIT:
If another dimension is introduced (in this case p), something like:
select dn.p, dn.val as dnval, min(up.val) as upval
from mytable up
join mytable dn
on dn.val <= up.val
and dn.p = up.p
where not exists (select 1 from mytable a where a.val = up.val + 1 and a.p = up.p)
and not exists (select 1 from mytable b where b.val = dn.val - 1 and b.p = dn.p)
group by dn.p, dn.val
order by dn.p, dn.val;
can be used, see Fiddle2

How to select last and last but one records

I have a table with 3 columns id, type, value like in image below.
What I'm trying to do is to make a query to get the data in this format:
type previous current
month-1 666 999
month-2 200 15
month-3 0 12
I made this query but it gets just the last value
select *
from statistics
where id in (select max(id) from statistics group by type)
order
by type
EDIT: Live example http://sqlfiddle.com/#!9/af81da/1
Thanks!
I would write this as:
select s.*,
(select s2.value
from statistics s2
where s2.type = s.type
order by id desc
limit 1, 1
) value_prev
from statistics s
where id in (select max(id) from statistics s group by type) order by type;
This should be relatively efficient with an index on statistics(type, id).
select
type,
ifnull(max(case when seq = 2 then value end),0 ) previous,
max( case when seq = 1 then value end ) current
from
(
select *, (select count(*)
from statistics s
where s.type = statistics.type
and s.id >= statistics.id) seq
from statistics ) t
where seq <= 2
group by type

How to select next record value in sql result

EDIT: To clarify there are many users and each user has many records, this is a log table of user activities,
how to find the timestamp difference every record and subsequent record that satisfies some condition ,
for example assuming the table is something like this
| id |u_id| .. | timestamp |
|----|----|----|--------------------|
| 50 | 1 | .. | 2014-04-22 15:35:44|
| 90 | 2 | .. | 2014-04-22 13:35:44|
| .. | .. | .. | ..... |
How do I find the time difference between every record and the next record for only one user id ?
Assuming that you want to do this for all users, the easiest way is to use variables:
select t.*,
if(u_id = #u_id, timediff(`timestamp`, #timestamp), NULL) as diff,
#timestamp := `timestamp`, #u_id := u_id
from table t cross join
(select #timestamp := 0, #u_id := 0) var
order by u_id, timestamp;
It is important that you explicitly order the records to be sure that the processing occurs in sequential order.
Try
select timediff(`timestamp`, #lasttime),
#lasttime := `timestamp`
from your_table
cross join (select #lasttime := 0) d
where u_id = 1
order by id
There are a couple of ways you can do this, the first would be to use a correlated subquery:
SELECT T.id,
T.u_id,
timestamp,
( SELECT T2.timestamp
FROM T AS T2
WHERE T2.u_id = T.u_id
AND T2.timestamp > T.timestamp
ORDER BY T2.Timestamp
LIMIT 1
) AS NextTimeStamp
FROM T;
Or you could do this using JOIN.
SELECT T.id,
T.u_id,
T.timestamp,
T2.timestamp AS NextTimeStamp
FROM T
LEFT JOIN T AS T2
ON T2.u_id = T.u_id
AND T2.timestamp > T.timestamp
LEFT JOIN T AS T3
ON T3.u_id = T.u_id
AND T3.timestamp > T.timestamp
AND T3.timestamp < T2.timestamp
WHERE T3.id IS NULL;
Which one is best will depend on your actual requirements, amount of data, and indexes.

MySql - How get value in previous row and value in next row? [duplicate]

This question already has answers here:
How to get next/previous record in MySQL?
(23 answers)
Closed 4 years ago.
I have the following table, named Example:
id(int 11) //not autoincriment
value (varchar 100)
It has the following rows of data:
0 100
2 150
3 200
6 250
7 300
Note that id values are not contiguous.
I've written this SQL so far:
SELECT * FROM Example WHERE id = 3
However, I don't know how to get the value of previous id and value of the next id...
Please help me get previous value and next value if id = 3 ?
P.S.: in my example it will be: previous - 150, next - 250.
Select the next row below:
SELECT * FROM Example WHERE id < 3 ORDER BY id DESC LIMIT 1
Select the next row above:
SELECT * FROM Example WHERE id > 3 ORDER BY id LIMIT 1
Select both in one query, e.g. use UNION:
(SELECT * FROM Example WHERE id < 3 ORDER BY id DESC LIMIT 1)
UNION
(SELECT * FROM Example WHERE id > 3 ORDER BY id LIMIT 1)
That what you mean?
A solution would be to use temporary variables:
select
#prev as previous,
e.id,
#prev := e.value as current
from
(
select
#prev := null
) as i,
example as e
order by
e.id
To get the "next" value, repeat the procedure. Here is an example:
select
id, previous, current, next
from
(
select
#next as next,
#next := current as current,
previous,
id
from
(
select #next := null
) as init,
(
select
#prev as previous,
#prev := e.value as current,
e.id
from
(
select #prev := null
) as init,
example as e
order by e.id
) as a
order by
a.id desc
) as b
order by
id
Check the example on SQL Fiddle
May be overkill, but it may help you
please try this sqlFiddle
SELECT value,
(SELECT value FROM example e2
WHERE e2.value < e1.value
ORDER BY value DESC LIMIT 1) as previous_value,
(SELECT value FROM example e3
WHERE e3.value > e1.value
ORDER BY value ASC LIMIT 1) as next_value
FROM example e1
WHERE id = 3
Edit: OP mentioned to grab value of previous id and value of next id in one of the comments so the code is here SQLFiddle
SELECT value,
(SELECT value FROM example e2
WHERE e2.id < e1.id
ORDER BY id DESC LIMIT 1) as previous_value,
(SELECT value FROM example e3
WHERE e3.id > e1.id
ORDER BY id ASC LIMIT 1) as next_value
FROM example e1
WHERE id = 3
SELECT *,
(SELECT value FROM example e1 WHERE e1.id < e.id ORDER BY id DESC LIMIT 1 OFFSET 0) as prev_value,
(SELECT value FROM example e2 WHERE e2.id > e.id ORDER BY id ASC LIMIT 1 OFFSET 0) as next_value
FROM example e
WHERE id=3;
And you can place your own offset after OFFSET keyword if you want to select records with higher offsets for next and previous values from the selected record.
Here's my solution may suit you:
SELECT * FROM Example
WHERE id IN (
(SELECT MIN(id) FROM Example WHERE id > 3),(SELECT MAX(id) FROM Example WHERE id < 3)
)
Demo: http://sqlfiddle.com/#!9/36c1d/2
A possible solution if you need it all in one row
SELECT t.id, t.value, prev_id, p.value prev_value, next_id, n.value next_value
FROM
(
SELECT t.id, t.value,
(
SELECT id
FROM table1
WHERE id < t.id
ORDER BY id DESC
LIMIT 1
) prev_id,
(
SELECT id
FROM table1
WHERE id > t.id
ORDER BY id
LIMIT 1
) next_id
FROM table1 t
WHERE t.id = 3
) t LEFT JOIN table1 p
ON t.prev_id = p.id LEFT JOIN table1 n
ON t.next_id = n.id
Sample output:
| ID | VALUE | PREV_ID | PREV_VALUE | NEXT_ID | NEXT_VALUE |
|----|-------|---------|------------|---------|------------|
| 3 | 200 | 2 | 150 | 4 | 250 |
Here is SQLFiddle demo
This query uses a user defined variable to calculate the distance from the target id, and a series of wrapper queries to get the results you want. Only one pass is made over the table, so it should perform well.
select * from (
select id, value from (
select *, (#x := ifnull(#x, 0) + if(id > 3, -1, 1)) row from (
select * from mytable order by id
) x
) y
order by row desc
limit 3
) z
order by id
See an SQLFiddle
If you don't care about the final row order you can omit the outer-most wrapper query.
If you do not have an ID this has worked for me.
Next:
SELECT * FROM table_name
WHERE column_name > current_column_data
ORDER BY column_name ASC
LIMIT 1
Previous:
SELECT * FROM table_name
WHERE column_name < current_column_data
ORDER BY column_name DESC
LIMIT 1
I use this for a membership list where the search is on the last name of the member. As long as you have the data from the current record it works fine.

Checking for maximum length of consecutive days which satisfy specific condition

I have a MySQL table with the structure:
beverages_log(id, users_id, beverages_id, timestamp)
I'm trying to compute the maximum streak of consecutive days during which a user (with id 1) logs a beverage (with id 1) at least 5 times each day. I'm pretty sure that this can be done using views as follows:
CREATE or REPLACE VIEW daycounts AS
SELECT count(*) AS n, DATE(timestamp) AS d FROM beverages_log
WHERE users_id = '1' AND beverages_id = 1 GROUP BY d;
CREATE or REPLACE VIEW t AS SELECT * FROM daycounts WHERE n >= 5;
SELECT MAX(streak) AS current FROM ( SELECT DATEDIFF(MIN(c.d), a.d)+1 AS streak
FROM t AS a LEFT JOIN t AS b ON a.d = ADDDATE(b.d,1)
LEFT JOIN t AS c ON a.d <= c.d
LEFT JOIN t AS d ON c.d = ADDDATE(d.d,-1)
WHERE b.d IS NULL AND c.d IS NOT NULL AND d.d IS NULL GROUP BY a.d) allstreaks;
However, repeatedly creating views for different users every time I run this check seems pretty inefficient. Is there a way in MySQL to perform this computation in a single query, without creating views or repeatedly calling the same subqueries a bunch of times?
This solution seems to perform quite well as long as there is a composite index on users_id and beverages_id -
SELECT *
FROM (
SELECT t.*, IF(#prev + INTERVAL 1 DAY = t.d, #c := #c + 1, #c := 1) AS streak, #prev := t.d
FROM (
SELECT DATE(timestamp) AS d, COUNT(*) AS n
FROM beverages_log
WHERE users_id = 1
AND beverages_id = 1
GROUP BY DATE(timestamp)
HAVING COUNT(*) >= 5
) AS t
INNER JOIN (SELECT #prev := NULL, #c := 1) AS vars
) AS t
ORDER BY streak DESC LIMIT 1;
Why not include user_id in they daycounts view and group by user_id and date.
Also include user_id in view t.
Then when you are queering against t add the user_id to the where clause.
Then you don't have to recreate your views for every single user you just need to remember to include in your where clause.
That's a little tricky. I'd start with a view to summarize events by day:
CREATE VIEW BView AS
SELECT UserID, BevID, CAST(EventDateTime AS DATE) AS EventDate, COUNT(*) AS NumEvents
FROM beverages_log
GROUP BY UserID, BevID, CAST(EventDateTime AS DATE)
I'd then use a Dates table (just a table with one row per day; very handy to have) to examine all possible date ranges and throw out any with a gap. This will probably be slow as hell, but it's a start:
SELECT
UserID, BevID, MAX(StreakLength) AS StreakLength
FROM
(
SELECT
B1.UserID, B1.BevID, B1.EventDate AS StreakStart, DATEDIFF(DD, StartDate.Date, EndDate.Date) AS StreakLength
FROM
BView AS B1
INNER JOIN Dates AS StartDate ON B1.EventDate = StartDate.Date
INNER JOIN Dates AS EndDate ON EndDate.Date > StartDate.Date
WHERE
B1.NumEvents >= 5
-- Exclude this potential streak if there's a day with no activity
AND NOT EXISTS (SELECT * FROM Dates AS MissedDay WHERE MissedDay.Date > StartDate.Date AND MissedDay.Date <= EndDate.Date AND NOT EXISTS (SELECT * FROM BView AS B2 WHERE B1.UserID = B2.UserID AND B1.BevID = B2.BevID AND MissedDay.Date = B2.EventDate))
-- Exclude this potential streak if there's a day with less than five events
AND NOT EXISTS (SELECT * FROM BView AS B2 WHERE B1.UserID = B2.UserID AND B1.BevID = B2.BevID AND B2.EventDate > StartDate.Date AND B2.EventDate <= EndDate.Date AND B2.NumEvents < 5)
) AS X
GROUP BY
UserID, BevID