SELECT without duplicates unless interupted by other values - mysql

Question

Given the following data:
name | temp
-----------
hoi | 15
hoi | 15
hoi | 16
hoi | 15
hej | 13
hoi | 13
I would like to select the data in the given two columns without duplicates, However I do want to keep duplicates that are duplicates if they where interrupted by another value:
name | temp
-----------
hoi | 15 // selected
hoi | 15 // ignored duplicate
hoi | 15 // ignored duplicate
hoi | 16 // selected
hoi | 15 // selected because while not being unique it follows a different value
hoi | 15 // ignored duplicate
hej | 13 // selected
hoi | 13 // selected
hoi | 13 // ignored duplicate
hoi | 14 // selected
hoi | 13 // selected because while not being unique it follows a different value
This question was hard to formulate for me given English is not my native tongue, Feel free to edit the question or ask for clarifications.
Edit:
There is an id field and a datetime field.
Edit 2:
I use mySQL 5.7

Answer 1

Since you are using MySQL 5.7, which doesn't support analytical functions, you will need to use variables to store the values of temp and name, from the previous row:
SELECT t.ID,
t.Name,
t.Temp
FROM ( SELECT t.*,
IF(#temp = t.temp AND #name = t.Name, 1, 0) AS IsDuplicate,
#temp:= t.temp,
#name:= t.Name
FROM YourTable AS t
CROSS JOIN (SELECT #temp := 0, #name := '') AS v
ORDER BY t.ID
) AS t
WHERE t.IsDuplicate = 0
ORDER BY ID;
Example on DB<>Fiddle
The key parts are (not in the order in which they appear, but in the order in which it is logical to think about it).
(1) Initialise the variables, and order by ID (or whatever field(s) you like) to ensure variables are assigned in the correct order
CROSS JOIN (SELECT #temp := 0, #name := '') AS v
ORDER BY t.ID
(2) Check if the values stored in the variables matches the current row, and flag with a 1 or a 0
IIF(#temp = t.temp AND #name = t.Name, 1, 0) AS IsDuplicate
(3) Assign the values of temp and name in the current row to the variables, so they can be checked against the next row:
#temp:= t.temp,
#name:= t.Name
(4) Remove duplicates from the final data set:
WHERE t.IsDuplicate = 0;
To go one further, you could change the IsDuplicate flag to be a group marker, and use GROUP BY, so you can find out how many records there were in total, while still not displaying duplicates:
SELECT MIN(ID) AS FirstID,
t.Name,
t.Temp,
COUNT(*) AS Records,
MAX(ID) AS LastID
FROM ( SELECT t.*,
#group:= IF(#temp = t.temp AND #name = t.Name, #group, #group + 1) AS GroupID,
#temp:= t.temp,
#name:= t.Name
FROM YourTable AS t
CROSS JOIN (SELECT #temp := 0, #name := '', #group:= 0) AS v
ORDER BY t.ID
) AS t
GROUP BY t.GroupID, t.Name, t.Temp
ORDER BY t.GroupID;
Example on DB<>Fiddle
This may be surplus to requirements, but it can be useful as you are able to extract a lot more information than when just identifying duplicate rows.
Finally if/when you upgrade to version 8.0 or newer, you will be able to use ROW_NUMBER(), or if you move to any other DBMS that supports ROW_NUMBER() (which is most nowadays), then you can use the following:
SELECT MIN(ID) AS FirstID,
t.Name,
t.Temp,
COUNT(*) AS Records,
MAX(ID) AS LastID
FROM ( SELECT t.*,
ROW_NUMBER() OVER(ORDER BY ID) -
ROW_NUMBER() OVER(PARTITION BY Temp, Name ORDER BY ID) AS GroupID
FROM YourTable AS t
ORDER BY t.ID
) AS t
GROUP BY t.GroupID, t.Name, t.Temp
ORDER BY t.GroupID;
Example on DB<>Fiddle

Answer 2

Generic Solution
You can use the following query to do this on any DBMS:
select nd.*
from dedup nd
inner join (
-- find the previous id for each id
select id, (select max(id) from dedup where id < o.id) prev_id
from dedup o
) id_to_prev on id_to_prev.id = nd.id
-- join with the prev row to check for dups
left join dedup d on d.id = id_to_prev.prev_id
and d.name = nd.name
and d.temp = nd.temp
where d.id is null -- if no prev row found with same name+temp, include this row
order by nd.id
SQL Fiddle: http://sqlfiddle.com/#!9/0584ca3/9

Answer 3

If You are using Oracle:
select name, temp from (
select id,
name,
temp,
lag(temp,1,-99999) over (order by id) as temp_prev
from table
order by id) t
where t.temp != t.temp_prev
might work for You (depending on Your Oracle version!), it uses the LAG analytics function to look into previous rows values, creates a temp table then filters it.

Answer 4

create table #temp (name varchar(3),temp int)
insert into #temp values ('hoi',15)
insert into #temp values ('hoi',15)
insert into #temp values ('hoi',15)
insert into #temp values ('hoi',16)
insert into #temp values ('hoi',15)
insert into #temp values ('hoi',15)
insert into #temp values ('hej',13)
insert into #temp values ('hoi',13)
insert into #temp values ('hoi',13)
insert into #temp values ('hoi',14)
insert into #temp values ('hoi',13)
;with FinalResult as (
select ROW_NUMBER()Over(partition by name,temp order by name) RowNumber,*
from #temp
)
select * from FinalResult where RowNumber =1
drop table #temp

Answer 5

You want to look at the previous row in order to decide whether to show a row or not. This would be easy with LAG, available as of MySQL 8. With MySQL 5.7 you need a correlated subquery with LIMIT instead to get the previous row.
select *
from mytable
where not (name, temp) <=>
(
select prev.name, prev.temp
from mytable prev
where prev.id < mytable.id
order by prev.id desc
limit 1
);
Demo: https://dbfiddle.uk/?rdbms=mysql_5.7&fiddle=4c775dbee12298cd93c5087d7085982f