Loading data from text file to mysql database by eliminating duplicates - mysql

Question

I want to load the data from text file to database, if data already exists i need to escape that data while loading.
I am using below query to load the data from text file to mysql data base.
"Load data infile 'F:/wbrdata.txt' into table wbrdatatable
fields terminated by ','
optionally enclosed by ""
lines terminated by '\r\n'
Ignore 1 lines (channel, time, pulserate, dwellid, targetid);"
It is appending the data to existing table data. I want to avoid the common data which is already exists(duplicates) in table & file while loading to the database.
How can i achieve this?
thank you
regards
sankar

Answer 1

Try to load text file into the temporary table (the same as target table), then remove duplicates from the temporary table and copy the rest to the target table.
Example (suppose that wbrdatatable_temp is a temporary table with all data from text file):
CREATE TABLE wbrdatatable(
id INT(11) NOT NULL AUTO_INCREMENT,
column1 VARCHAR(255) DEFAULT NULL,
PRIMARY KEY (id)
);
INSERT INTO wbrdatatable VALUES
(1, '111'),
(2, '222'),
(3, '333'),
(4, '444'),
(5, '555');
CREATE TABLE wbrdatatable_temp(
id INT(11) NOT NULL AUTO_INCREMENT,
column1 VARCHAR(255) DEFAULT NULL,
PRIMARY KEY (id)
);
INSERT INTO wbrdatatable_temp VALUES
(1, '111'),
(2, '222'),
(10, '100'), -- new record that should be added
(11, '200'); -- new record that should be added
-- Copy only new records!
INSERT INTO wbrdatatable
SELECT t1.* FROM wbrdatatable_temp t1
LEFT JOIN wbrdatatable t2
ON t1.id = t2.id AND t1.column1 = t2.column1
WHERE t2.id IS NULL;
-- Test result
SELECT * FROM wbrdatatable;
+----+---------+
| id | column1 |
+----+---------+
| 1 | 111 |
| 2 | 222 |
| 3 | 333 |
| 4 | 444 |
| 5 | 555 |
| 10 | 100 | -- only new record is added
| 11 | 200 | -- only new record is added
+----+---------+

Answer 2

Try this Logic.
1. Upload Text File Data
2. Check record using select statement on your database
if(recordexist==true)
save
else
not save
Regards