Using LIMIT on MySql deletion across a two column duplicate - mysql

Question

I have a large MySql table from which I need to delete duplicates - to qualify as a duplicate, a row much match another row on two columns:
SELECT * FROM JwDistanceSurnames n1, JwDistanceSurnames n2
WHERE n1.JwDistanceSurnameId > n2.JwDistanceSurnameId
AND n1.Surname1 = n2.Surname1
AND n1.Surname2 = n2.Surname2
LIMIT 1000;
Because it is a large table, I'd like to do it in batches. My understanding is that I ought to be able to use LIMIT to achieve this. However, this does not execute, citing a syntax error:
DELETE n1 FROM JwDistanceSurnames n1, JwDistanceSurnames n2
WHERE n1.JwDistanceSurnameId > n2.JwDistanceSurnameId
AND n1.Surname1 = n2.Surname1
AND n1.Surname2 = n2.Surname2
LIMIT 1000;
What's the error? Is it not possible to use this simple approach to batching here?
MCVE:
CREATE TABLE `JwDistanceSurnames` (
`JwDistanceSurnameId` int(11) NOT NULL AUTO_INCREMENT,
`Surname1` varchar(999) DEFAULT NULL,
`Surname2` varchar(999) DEFAULT NULL,
`JwScore` double NOT NULL,
PRIMARY KEY (`JwDistanceSurnameId`),
KEY `Surname1` (`Surname1`),
KEY `Surname2` (`Surname2`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
INSERT INTO `JwDistanceSurnames`
(`JwDistanceSurnameId`, `Surname1`, `Surname2`, `JwScore`)
VALUES (null,'williamsom' ,'williamson' ,0.959999999999998);
Repeat the insert a few times. Then run the delete. The expected output is a single row, with the given values. Which of the rows kept is not important.
The error is:
Error Code: 1064. You have an error in your SQL syntax; check the
manual that corresponds to your MySQL server version for the right
syntax to use near 'ORDER BY n1.JwDistanceSurnameId LIMIT 1000' at
line 5

Answer 1

From this SO question, it appears that LIMIT cannot be used in a DELETE statement when more than one table is being referenced. One trick around this is to use LIMIT in a subquery to identify records for deletion, and then join back to the target table:
DELETE t1
FROM JwDistanceSurnames t1
INNER JOIN
(
SELECT n1.JwDistanceSurnameId
FROM JwDistanceSurnames n1
INNER JOIN JwDistanceSurnames n2
ON n1.JwDistanceSurnameId > n2.JwDistanceSurnameId
WHERE n1.Surname1 = n2.Surname1 AND n1.Surname2 = n2.Surname2
ORDER BY <some_column> -- IMPORTANT! without this you may get random records
LIMIT 1000
) t2
ON t1.JwDistanceSurnameId = t2.JwDistanceSurnameId;
So the subquery labelled t2 uses LIMIT to identify batches of 1000 records at a time for deletion, and then we use another join to actually label those target records.
Also note that using LIMIT without ORDER BY is not really a well-defined thing, because SQL tables are modelled on unordered sets of records. If you have some business logic determining which order the batches should be deleted, then consider adding an ORDER BY clause (unless it truly does not matter, which would seem unlikely to me).

Answer 2

I think you can use another way for find of dublicates
SELECT n.*
FROM JwDistanceSurnames n
JOIN
(
SELECT Surname1,Surname2,MIN(JwDistanceSurnameId) min_JwDistanceSurnameId
FROM JwDistanceSurnames
GROUP BY Surname1,Surname2
) l
ON n.Surname1=n.Surname1 AND n.Surname2=n.Surname2 AND n.JwDistanceSurnameId>l.min_JwDistanceSurnameId