how to delete every record except one per hour - mysql

Question

I have a mysql table with millions of sensor records with the following structure:
datanumber (auto increment),
stationid (int),
sensortype (int),
measuredate (datetime),
data (medtext)
each stations adds a record every 2-10 minute per sensortype (2-5 sensors)
I would like to keep only one record per hour, per sensor, per station
and this too only if measuredate is older than 1 year.
I understand how to select data older than one year but I have no clue on how to delete rows except one for each hour. It does not really matter if it's the first, last or a random value which is kept at each hour. I also do not need to calculate average values or something, just strip down the amount of records stored

Answer 1

You should be able to do something like
Select * from observations where <old> group by sensortype, stationid, extract(year_month, measure_date), extract(day_hour, measure_date);
group_by will collapse the records in each group into one. You could select this into a new table if you want.
If you need to actually delete all the redundant old records, just select the datanumbers using the above query, and then delete all records NOT IN(<those ids>).

Answer 2

If you are going to be deleting a very large number of rows, then one approach recommended by the MySQL docs is to select the rows you want to retain into a temporary table, and then perform an atomic table renaming. Maybe like this:
INSERT INTO
sensordata_squeezed
SELECT
datanumber,
stationid,
sensortype,
measuredate,
data
FROM
sensordata
WHERE
measuredate < DATE_SUB(CURDATE(), INTERVAL 1 YEAR)
GROUP BY
DATE_ADD(DATE(measuredate), INTERVAL HOUR(measuredate) HOUR),
stationid,
sensortype
UNION ALL
SELECT
datanumber,
stationid,
sensortype,
measuredate,
data
FROM
sensordata
WHERE
measuredate >= DATE_SUB(CURDATE(), INTERVAL 1 YEAR)
;
RENAME TABLE
sensordata TO sensordata_old,
sensordata_squeezed TO sensordata
;
DROP TABLE sensordata_old
;
Note well: that relies on MySQL's documented behavior with respect to selecting columns from an aggregate query that are neither grouping columns nor aggregate functions of the groups: it chooses an indeterminate value from each group. (This is an extension to standard SQL.) I am supposing that within each group, all the nonaggregated column values will come from the same row; you should check because that part is not documented, and this approach depends on that to maintain data integrity.
This approach allows you to avoid both large, expensive joins, and large numbers of subqueries.
Do note that however you do this, you are going to have to work around issues of how to avoid losing data that comes in while this operation is running, as it is likely to take a long time.

Answer 3

This would be a lead-pipe cinch if we could use row_number over( ... ) but a solution for MySQL is not difficult. For problems like this, look to see if we can query a list of just the rows we want to delete. That sounds easy enough. First, we want to have a list of each hour of each day and the first (least) entry for that hour:
select Date( MeasureDate ) TheDate, Hour( MeasureDate ) TheHour, Min( MeasureDate ) MinTime
from T
group by TheDate, TheHour;
So we just have to join the table back to this result set:
select T.*
from T
join(
select Date( MeasureDate ) TheDate, Hour( MeasureDate ) TheHour, Min( MeasureDate ) MinTime
from T
group by TheDate, TheHour
) as T1
on T1.MinTime = T.MeasureDate
This gives us all the rows we want to keep. So use a left join to invert the results:
select T.*
from T
left join(
select Date( MeasureDate ) TheDate, Hour( MeasureDate ) TheHour, Min( MeasureDate ) MinTime
from T
group by TheDate, TheHour
) as T1
on T1.MinTime = T.MeasureDate
where T1.MinTime is null;
Change the select to delete et viola:
delete TDel
from T TDel
left join(
select Date( MeasureDate ) TheDate, Hour( MeasureDate ) TheHour, Min( MeasureDate ) MinTime
from T
group by TheDate, TheHour
) as T1
on T1.MinTime = TDel.MeasureDate
where T1.MinTime is null;
You can add other fields such as SensorType as appropriate to keep first entry of each hour per sensor or however you want to tune it. SqlFiddle