I have a question regarding interval calculation with timestamps:
Let´s assume I have a table with IDs and timestamps:
Table tab1
tab1.id deviceid timestamp
------- -------- ----------
1 15 2013-01-01 14:57:54
2 15 2013-01-01 14:58:09
3 23 2013-01-01 14:58:10
4 15 2013-01-01 14:58:30
What I want to do is to check if the intervals between the entries are inside or outside a specific value. Let´s assume 15 sec is the value:
The interval between 1 and 2 is OK, between 2 and 3 not OK.
Any hints how to solve this without using a temporary table and stored procedure etc.?
Thanks in advance and kind regards
solick
EDIT: Updated the table. There are timestamps from other devices in between the entries.
You can start with this. This will join the table against itself and calculate the +1 id to calculate the difference in seconds.
SELECT a.id,
TIME_TO_SEC(TIMEDIFF(b.timestamp, a.timestamp)) AS time_diff
FROM tab1 a
LEFT JOIN tab1 b ON b.id = a.id + 1
WHERE b.id IS NOT NULL
Result
| ID | TIME_DIFF |
------------------
| 1 | 15 |
| 2 | 21 |
SQL Fiddle
Solution:
njk give me the right hint, but at the end it was necessary to use a stored procedure and temoprary table:
Important here: Temoprary tables cannot be used more than once within a query, therefore i needed to copy the temporary table:
BEGIN
DROP TEMPORARY TABLE IF EXISTS tmptable1;
DROP TEMPORARY TABLE IF EXISTS tmptable2;
CREATE TEMPORARY TABLE tmptable1 (tmpid INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY)
SELECT * FROM tbl_source WHERE tbl_source.Id = did;
CREATE TEMPORARY TABLE tmptable2 AS (SELECT * FROM tmptable1);
SELECT a.tmpid, a.timestamp, TIME_TO_SEC(TIMEDIFF(b.timestamp, a.timestamp)) AS time_diff
FROM tmptable1 a LEFT JOIN tmptable2 b
ON b.tmpid = a.tmpid +1
WHERE b.tmpid IS NOT NULL;
DROP TEMPORARY TABLE IF EXISTS tmptable1;
DROP TEMPORARY TABLE IF EXISTS tmptable2;
END
Related
I know there is a ton of same questions about finding and removing duplicate values in mySQL but my question is a bit different:
I have a table with columns as ID, Timestamp and price. A script scrapes data from another webpage and saves it in the database every 10 seconds. Sometimes data ends up like this:
| id | timestamp | price |
|----|-----------|-------|
| 1 | 12:13 | 100 |
| 2 | 12:14 | 120 |
| 3 | 12:15 | 100 |
| 4 | 12:16 | 100 |
| 5 | 12:17 | 110 |
As you see there are 3 duplicated values and removing the price with ID = 4 will shrink the table without damaging data integrity. I need to remove continuous duplicated records except the first one (which has the lowest ID or Timestamp).
Is there a sufficient way to do it? (there is about a million records)
I edited my scraping script so it checks for duplicated price before adding it but I need to shrink and maintain my old data.
Since MySQL 8.0 you can use window function LAG() in next way:
delete tbl.* from tbl
join (
-- use lag(price) for get value from previous row
select id, lag(price) over (order by id) price from tbl
) l
-- join rows with same previous price witch will be deleted
on tbl.id = l.id and tbl.price = l.price;
fiddle
I am just grouping based on price and filtering only one record per group.The lowest id gets displayed.Hope the below helps.
select id,timestamp,price from yourTable group by price having count(price)>0;
My query is based on #Tim Biegeleisen one.
-- delete records
DELETE
FROM yourTable t1
-- where exists an older one with the same price
WHERE EXISTS (SELECT 1
FROM yourTable t2
WHERE t2.price = t1.price
AND t2.id < t1.id
-- but does not exists any between this and the older one
AND NOT EXISTS (SELECT 1
FROM yourTable t3
WHERE t1.price <> t3.price
AND t3.id > t2.id
AND t3 < t1.id));
It deletes records where exists an older one with same price but does not exists any different between
It could be checked by timestamp column if id column is not numeric and ascending.
I have multiple SELECT statements that all return the same columns but may return different resultsets. Is there any way to select all rows that are in all resultsets on database level?
E.g.
|---------------------|------------------|---------|
| ID | Name | Age |
|---------------------|------------------|---------|
| 1 | Paul | 50 |
| 2 | Peter | 40 |
| 3 | Frank | 20 |
| 4 | Pascal | 60 |
|---------------------|------------------|---------|
SELECT 1
SELECT name FROM table WHERE age > 40
Result: Paul, Pascal
SELECT 2
SELECT name FROM table where name like 'P%'
Result: Paul, Peter, Pascal
SELECT 3
SELECT name FROM table where id > 3
Result: Pascal
EDIT: This is a very simplified example of my problem. The statements can get very complex (joins over multiple tables), so a simple AND in the WHERE part is not the final solution.
The result should be Pascal. What I am looking for is something like a "reverse UNION".
Alternatively it would be possible to achieve that programatically (NodeJS), but I would like to avoid to iterate over all resultsets, because they might be quite huge.
Thanks in advance!
Is there any way to select all rows that are in all resultsets?
You seem to want and:
select name
from table
where age > 40 and name like 'P%' and id < 3
If using AND between the WHERE conditions is not possible, you could use multiple IN expressions on subqueries using your initial queries.
SELECT name
FROM table
WHERE id IN (SELECT id FROM table WHERE age > 40)
AND id IN (SELECT id FROM table where name like 'P%')
AND id IN (SELECT id FROM table where id < 3)
If you have different result sets and you want to see the intersection, you can use join:
select q1.id
from (<query 1>) q1 join
(<query 2>) q2
on q1.id = q2.id join
(<query 3>) q3
on q1.id = q3.id;
That said, I think GMB has the most concise answer to the question that you actually asked.
If your statements are complex, what you could do is to use a procedure where each of the statements put the matching id's into a temp table. Then select those rows where id's match the number of statements. This will also most likely be more efficient than one huge query with all complex statements combined into one.
create procedure sp_match_all()
begin
drop temporary table if exists match_tmp;
create temporary table match_tmp (
id int
);
insert into match_tmp
SELECT id FROM table WHERE age > 40;
insert into match_tmp
SELECT id FROM table where name like 'P%';
insert into match_tmp
SELECT id FROM table where id < 3;
select t.name
from table t
join (
select id
from match_tmp
group by id
having count(*)=3
) q on q.id=t.id;
drop temporary table match_tmp;
end
I've seen many questions along this issue, but can't get this to work.
I want to UPDATE multiple columns in a table (but will start with one) based upon a calculated value from the same table.
It is a list of transactions per customer, per month.
TransID | Cust | Month | Value | PastValue | FutureValue
1 | 1 | 2018-01-01 | 45 |
2 | 1 | 2018-02-01 | 0 |
3 | 1 | 2018-03-01 | 35 |
4 | 1 | 2018-04-01 | 80 |
.
UPDATE tbl_transaction a
SET PrevMnthValue =
(SELECT COUNT(TransactionID) FROM tbl_transaction b WHERE b.Cust=a.Cust AND b.Month<a.Month)
But we get the dreaded 'Can't update a table using a where with a subquery of the same table).
I've tried to nest the subquery as this has been touted as a workaround:
UPDATE tbl_transactions a
SET
PastValue =
(
SELECT CNT FROM
(
SELECT
COUNT(TransactionID) AS CNT
FROM tbl_transactions b
WHERE
b.CustomerRef=a.CustomerRef AND b.Month<a.Month
) x
),
FutureValue =
(
SELECT CNT FROM
(
SELECT
COUNT(TransactionID) AS CNT
FROM tbl_transactions b
WHERE
b.CustomerRef=a.CustomerRef AND b.Month>a.Month
) x
)
But I get an UNKNOWN a.CustomerRef in WHERE clause. Where am I going wrong?
You can't update and read from one table at the same time.
MySQL documentation tell about it
You cannot update a table and select from the same table in a subquery.
At first you must select necessary data and save them to somewhere, for example to temporary table
CREATE TEMPORARY TABLE IF NOT EXISTS `temp` AS (
SELECT
COUNT(`TransactionID`) AS CNT,
`CustomerRef`,
`Month`
FROM `tbl_transactions`
GROUP BY `Custom,erRef`, `Month`
);
After it, you can use JOIN statement for update table
UPDATE `tbl_transactions` RIGTH
JOIN `temp` ON `temp`.`CustomerRef` = `tbl_transactions`.`CustomerRef`
AND `temp`.`Month` < `tbl_transactions`.`Month`
SET `tbl_transactions`.`PastValue` = `temp`.`cnt`
UPDATED: if you want to update several columns by different condition you can combine temporary table, UPDATE + RIGHT JOIN and CASE statement. For example:
UPDATE `tbl_transactions`
RIGTH JOIN `temp` ON `temp`.`CustomerRef` = `tbl_transactions`.`CustomerRef`
SET `tbl_transactions`.`PastValue` = CASE
WHEN `temp`.`Month` < `tbl_transactions`.`Month` THEN `temp`.`cnt`
ELSE `tbl_transactions`.`PastValue`
END,
`tbl_transactions`.`FutureValue` = CASE
WHEN `temp`.`Month` > `tbl_transactions`.`Month` THEN `temp`.`cnt`
ELSE `tbl_transactions`.`FutureValue`
END
You can try below
UPDATE tbl_transactions a
Join
( SELECT CustomerRef,COUNT(TransactionID) AS CNT FROM tbl_transactions b
group by CustomerRef)x
SET PastValue = CNT
WHERE x.CustomerRef=a.CustomerRef AND x.Month<a.Month
i have a large MySQL Database with more than 1 Million rows. How can i find the missing eid's?
+----+-----+
| id | eid |
+----+-----+
| 1 | 1 |
+----+-----+
| 2 | 2 |
+----+-----+
| 3 | 4 |
+----+-----+
I like to list all missing eid's, the 3 in this example. I've tried many things but everything what i do need to much time.
I hope someone can help me.
Thanks
You can use NOT EXISTS to find the required rows.
create table t(id integer, eid integer);
insert into t values(1,1);
insert into t values(2,2);
insert into t values(3,4);
SELECT id
FROM t a
WHERE NOT EXISTS
( SELECT 1
FROM t b
WHERE b.eid = a.id );
or use NOT IN:
SELECT ID
FROM t
WHERE ID NOT IN
(SELECT EID
FROM t);
produces:
| id |
|----|
| 3 |
Try the below query
SELECT ID FROM table WHERE ID NOT IN(SELECT EID FROM table );
Finding duplicate numbers is easy:
select id, count() from sequence
group by id
having count() > 1;
In this case there are no duplicates, since I’m not concentrating on that in this post (finding duplicates is straightforward enough that I hope you can see how it’s done). I had to scratch my head for a second to find missing numbers in the sequence, though. Here is my first shot at it:
select l.id + 1 as start
from sequence as l
left outer join sequence as r on l.id + 1 = r.id
where r.id is null;
The idea is to exclusion join against the same sequence, but shifted by one position. Any number with an adjacent number will join successfully, and the WHERE clause will eliminate successful matches, leaving the missing numbers. Here is the result:
https://www.xaprb.com/blog/2005/12/06/find-missing-numbers-in-a-sequence-with-sql/
if you want a lighter way to search millions of rows of data,
I was try for search in more than 23 millions rows with old CPU (12.6Gb data need about 1gb of free ram):
Affected rows: 0 Found rows: 346.764 Warnings: 0 Duration for 2 queries: 00:04:48.0 (+ 2,656 sec. network)
SET #idBefore=0, #st=0,#diffSt=0,#diffEnd=0;
SELECT res.idBefore `betweenID`, res.ID `andNextID`
, res.startEID, res.endEID
, res.diff `diffEID`
-- DON'T USE this missingEIDfor more than a thousand of rows
-- this is just for sample view
, GROUP_CONCAT(b.aNum) `missingEID`
FROM (
SELECT
#idBefore `idBefore`
, #idBefore:=(a.id) `ID`
, #diffSt:=(#st) `startEID`
, #diffEnd:=(a.eid) `endEID`
, #st:=a.eid `end`
, #diffEnd-#diffSt-1 `diff`
FROM eid a
ORDER BY a.ID
) res
-- DON'T USE this integers for more than a thousand of rows
-- this is just for sample view
CROSS JOIN (SELECT a.ID + (b.ID * 10) + (c.ID * 100) AS aNum FROM integers a, integers b, integers c) b
WHERE res.diff>0 AND b.aNum BETWEEN res.startEID+1 AND res.endEID-1
GROUP BY res.ID;
check out this http://sqlfiddle.com/#!9/33deb3/9
and this is for missing ID http://sqlfiddle.com/#!9/3ea00c/9
I have two rows each of which contain a week a day and an event. An auto increment primary key is used to distinguish the rows.
Here is an example:
ID Week Day event
-------------------------------
1 | 1 | 2 | house keeping
2 | 2 | 3 | house viewing
What i want to do is swap the week and day of the two rows specified so that it looks like this:
ID Week Day event
-------------------------------
1 | 2 | 3 | house keeping
2 | 1 | 2 | house viewing
But the Id must remain the same
Ive been reading through other peoples posts and found this solution which uses temporary variables to swap only one columns values from each row.
UPDATE my_table SET a=#tmp:=a, a=b, b=#tmp;
Could anyone help me swap two columns instead of just the one?
thanks
I assume you have just 2 rows in your table.
If not, you need to modify slightly the JOIN conditions.
Here is one possible approach.
CREATE TEMPORARY TABLE T(ID int, Week int, Day int)
INSERT INTO T(ID, Week, Day)
SELECT ID, Week, Day from TableName;
UPDATE TableName t1
JOIN T t2 on t1.ID <> t2.ID
SET
t1.Week = t2.Week,
t1.Day = t2.Day;
DROP TEMPORARY TABLE T;
And here is a better one.
UPDATE
tablename AS t1
JOIN tablename AS t2 ON
( t1.id <> t2.id )
SET
t1.week = t2.week,
t2.week = t1.week,
t1.day = t2.day,
t2.day = t1.day;