mysql query to efficiently remove duplicates - mysql

Question

Hi folks and thanks for reading
I have a quiz feature on my site which stores a score, username and ip address as the most important columns. I currently have a horrible series of views bringing back the high scores based on the criteria I need which are...
Lowest score first but...only the lowest score for each Quiz user.
The complexity lies if the user has changed ip, i.e. keeps the same username but has a different ip OR if the user keeps the same IP address but changes user name.
It's easier to explain with an example.
First visitor has 4 entries but from 3 different IP Addresses
Second user from 2 IP Addresses
Third user using one IP Address but using 3 Usernames
Table with VALUES(UserID, IPA, Score)
User 1, IP1, 13
User 1, IP1, 20
User 1, IP2, 30
User 1, IP3, 10
User 2, IP4, 20
User 2, IP5, 22
User 2, IP5, 15
User 3, IP6, 12
User 3, IP6, 20
User 4, IP6, 15
User 5, IP6, 11
The highscore query would present you with
User 1, IP3, 10
User 5, IP6, 11
User 2, IP5, 15
The score value is highly unlikely to be duplicated but I guess it is possible. The figures above are simplified to explain my conundrum!
Can anyone suggest an efficient way of removing these duplicates as my table is now over 15,000 records and the views are creaking!
Many thanks.

Answer 1

To identify occurrences of duplicate (UserID,IPA) tuples is pretty straightforward:
SELECT s.UserID
, s.IPA
FROM mytable s
GROUP
BY s.UserID
, s.IPA
HAVING COUNT(1) > 1
To get the lowest score, you could add MIN(s.Score) to the select list.
Deleting duplicates is a little more difficult, in that you don't seem to have any guarantee of uniqueness. Some will recommend that you copy the rows you want to keep out to a separate table, and then either swap the tables with renames, or truncate the original table and reload from the new table. (That usually turns out to be the most efficient approach.)
CREATE TABLE newtable LIKE mytable ;
INSERT INTO newtable (UserID,IPA,Score)
SELECT s.UserID
, s.IPA
, MIN(Score) AS Score
FROM mytable s
GROUP
BY s.UserID
, s.IPA ;
If you want to identify duplicates by just UserID, the same approach can work. If it isn't important that the IPA value comes from the row with the lowest score, it's a little easier. I can put together the query that gets the row that has the lowest score for the user.
If you want to delete rows from the existing table, without adding a unique identifier (like an AUTO_INCREMENT id column) on each row, that can be done too.
This will get you partway, deleting all rows for a given (UserID,IPA) that have a score higher than the lowest score:
DELETE t.*
FROM mytable t
JOIN ( SELECT s.UserID
, s.IPA
, MIN(s.Score)
FROM mytable s
GROUP
BY s.Userid
, s.IPA
) k
ON k.UserID = t.UserID
AND k.IPA = t.IPA
AND k.Score < t.Score
But that will still leave duplicate occurrences of duplicate (UserID,IPA,Score) tuples. Without some other column on the table that makes the row unique, it's a little more difficult to remove duplicates. (Again, a common technique is copy the rows you want to keep to another table, and either swap tables or reload the original table from the saved rows.
FOLLOWUP
Note that views (both stored and inline) can be expensive performancewise, with MySQL, since the views get materialized as temporary MyISAM tables (MySQL calls them "derived tables").
But correlated subqueries can be even more problematic on large sets.
So, choose your poison.
If there the table has an index ON (userID, Score, IPA) here's how I would get the resultset:
SELECT IF(#prev_user=t.UserID,#i:=#i+1,#i:=1) AS seq
, #prev_user := t.UserID AS UserID
, t.IPA
, t.Score
FROM mytable t
JOIN (SELECT #i := NULL, #prev_user := NULL) i
GROUP
BY t.UserID ASC
, t.Score ASC
, t.IPA ASC
HAVING seq = 1
This is taking advantage of some MySQL-specific features: user_variables and the guarantee that the GROUP BY will return a sorted resultset. (The EXPLAIN output will show "Using index", which means we avoid a sort operation, but the query will still create a derived table. We use the user_variables to identify the "first" row for each UserID, and the HAVING clause eliminates all but that first row.
test case:
create table mytable (UserID VARCHAR(6), IPA varchar(3), Score INT);
create index mytable_IX ON mytable (UserID, Score, IPA);
insert into mytable values ('User 1','IP1',13)
,('User 1','IP1',20)
,('User 1','IP2',30)
,('User 1','IP3',10)
,('User 2','IP4',20)
,('User 2','IP5',22)
,('User 2','IP5',15)
,('User 3','IP6',12)
,('User 3','IP6',20)
,('User 4','IP6',15)
,('User 5','IP6',11);
Another followup
To eliminate 'User 4' and 'User 5' from the resultset (it's not at all clear why you would want or need to do that. If it's because those users have only one row in the table, then you could add a JOIN to a subquery (inline view) that gets a list of UserID values where there is more than one row, like this:
SELECT IF(#prev_user=t.UserID,#i:=#i+1,#i:=1) AS seq
, #prev_user := t.UserID AS UserID
, t.IPA
, t.Score
FROM mytable t
JOIN ( SELECT d.UserID
FROM mytable d
GROUP
BY d.UserID
HAVING COUNT(1) > 1
) m
ON m.UserID = t.UserID
CROSS
JOIN (SELECT #i := NULL, #prev_user := NULL) i
GROUP
BY t.UserID ASC
, t.Score ASC
, t.IPA ASC
HAVING seq = 1