Detecting near duplicates above a threshold - mysql

Question

I want to be able to query a table for records I suspect may be nearly duplicates.
I've racked my brains but can't think where to begin with this one, so I've simplified the problem as much as possible, and came to ask here!
Here's my simplified table:
CREATE TABLE sales
(
`id1` int auto_increment primary key,
`amount` decimal(6,2),
`date` datetime
);
Here's some test values:
INSERT INTO sales
(`amount`, `date`)
VALUES
(10, '2013-05-15T11:11:00'),
(11, '2013-05-15T11:11:11'),
(20, '2013-05-15T11:22:00'),
(3, '2013-05-15T12:12:00'),
(4, '2013-05-15T12:12:12'),
(45, '2013-05-15T12:22:00'),
(4, '2013-05-15T12:24:00'),
(8, '2013-05-15T13:00:00'),
(9, '2013-05-15T13:01:00'),
(10, '2013-05-15T14:00:00');
The problem
I want to return sales above amount Y, that have neighbour sales above Y that recorded within X minutes of each other.
ie, from this data:
amt, date
(10, '2013-05-15T11:11:00'),
(11, '2013-05-15T11:11:11'),
(20, '2013-05-15T11:22:00'),
(3, '2013-05-15T12:12:00'),
(4, '2013-05-15T12:12:12'),
(45, '2013-05-15T12:22:00'),
(4, '2013-05-15T12:24:00'),
(8, '2013-05-15T13:00:00'),
(9, '2013-05-15T13:01:00'),
(10, '2013-05-15T14:00:00');
where #yVal = 5 and #xMins = 10
expected result would be:
(10, '2013-05-15T11:11:00'),
(11, '2013-05-15T11:11:11'),
(20, '2013-05-15T11:22:00'),
(8, '2013-05-15T13:00:00'),
(9, '2013-05-15T13:01:00'),
I've put the above into a fiddle: http://sqlfiddle.com/#!2/cf8fe
Any help will be greatly appreciated!

Answer 1

Try somthing like this:
SELECT DISTINCT s1.* FROM sales s1
LEFT JOIN sales s2
ON (s1.id1 != s2.id1
AND s1.amount >= s2.amount - #xVal AND s1.amount <= s2.amount + #xVal
AND s1.date >= DATE_SUB(s2.date, INTERVAL #xMins minute) AND s1.date <= DATE_ADD(s2.date, INTERVAL #xMins minute)
)
WHERE
s2.id1 is not null
Extends
Fix some errors
Result for your data looks like:
+-----+--------+---------------------+
| id1 | amount | date |
+-----+--------+---------------------+
| 1 | 10.00 | 2013-05-15 11:11:00 |
| 2 | 11.00 | 2013-05-15 11:11:11 |
| 4 | 3.00 | 2013-05-15 12:12:00 |
| 5 | 4.00 | 2013-05-15 12:12:12 |
| 8 | 8.00 | 2013-05-15 13:00:00 |
| 9 | 9.00 | 2013-05-15 13:01:00 |
+-----+--------+---------------------+
Extends 2
SELECT DISTINCT s1.* FROM sales s1
LEFT JOIN sales s2
ON (s1.id1 != s2.id1
AND s2.amount >= #xVal
AND s1.date >= DATE_SUB(s2.date, INTERVAL #xMins minute) AND s1.date <= DATE_ADD(s2.date, INTERVAL #xMins minute)
)
WHERE
s2.id1 is not null
AND s1.amount >= #xVal