Fast delete duplicate records in MySQL - mysql

Question

I'm trying to import very big SQL dump (around 37 million rows) into InnoDB table. There are tons of duplicates and what I want to achieve is, without changing actual dump want to prevent duplicate row insertion. The field email might have duplicates. I tried following: after importing whole dump into db I tried to execute following SQL:
set session old_alter_table=1;
ALTER IGNORE TABLE sample ADD UNIQUE (email);
But second query worked around 1 hour and then I just canceled this query.
What is proper way to get rid off duplicates?
I have couple of ideas:
Maybe before starting to import to make a table with unique index and while insertion to prevent duplicates without harming whole process?
Maybe after importing dump to select distinct email and to insert into another table?

Answer 1

From a .dump file
When importing, use -f for "force":
mysql -f -p < 2015-10-01.sql
This causes the import to continue after an error is encountered, which is useful in this case if you create the unique key constraint before importing.
From a .csv file
If you are using "LOAD DATA", use "IGNORE", e.g.:
LOAD DATA LOCAL INFILE 'somefile.csv' IGNORE
INTO TABLE some_db.some_tbl
FIELDS TERMINATED BY ';'
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(`somefield1`,`somefield2`);
According to the documentation:
If you specify IGNORE, rows that duplicate an existing row on a unique
key value are discarded.
This requires you to create the unique key constraint before importing, which will be fast on an empty table.

Answer 2

Edit the dump file as follows:
Modify the CREATE TABLE statement to add a unique key on the email field, or add an ALTER TABLE statement after it.
Find all the INSERT INTO sample statements, and change them to INSERT IGNORE INTO sample.
You could also do step 2 using a pipeline:
sed 's/INSERT INTO sample/INSERT IGNORE INTO sample/' sample_table.dump | mysql -u root -p sample_db
If the file is too big to edit to add the ALTER TABLE statement, I suggest you create the dump with the --no-create-info option to mysqldump, and create the table by hand (with the unique key) before loading the dump file.