I have a table called leads with duplicate records
Leads:
*account_id
*campaign_id
I want to remove all the duplicate account_id where campaign_id equal to "51"
For example, if account_id = 1991 appears two times in the table then remove the one with campaign_id = "51" and keep the other one.
You could use a delete join:
DELETE t1
FROM yourTable t1
INNER JOIN yourTable t2
ON t2.account_id = t1.account_id AND
t2.campaign_id <> 51
WHERE
t1.campaign_id = 51;
There's no problem to delete from a table provided that:
You use the correct syntax.
You have done a backup of the table BEFORE you do any deleting.
However, I would suggest a different method:
Create a new table based on the existing table:
CREATE TABLE mytable_new LIKE mytable;
Add unique constraint (or PRIMARY KEY) on column(s) you don't want to have duplicates:
ALTER TABLE mytable_new ADD UNIQUE(column1,[column2]);
Note: if you want to identify a combination of two (or more) columns as unique, place all the column names in the UNIQUE() separated by comma. Maybe in your case, the constraint would be UNIQUE(account_id, campaign_id).
Insert data from original table to new table:
INSERT IGNORE INTO mytable_new SELECT * FROM mytable;
Note: the IGNORE will insert only non-duplicate values that match with the UNIQUE() constraint. If you have an app that runs a MySQL INSERT query to the table, you have to update the query by adding IGNORE.
Check data consistency and once you're satisfied, rename both tables:
RENAME TABLE mytable TO mytable_old;
RENAME TABLE mytable_new TO mytable;
The best thing about this is that in case that if you see anything wrong with the new table, you still have the original table.
Changing the name of the tables only take less than a second, the probable issue here is that it might take a while to do the INSERT IGNORE if you have a large data.
Demo fiddle
DELETE t1
FROM yourTable t1
INNER JOIN yourTable t2
ON t2.account_id = t1.account_id AND
t2.campaign_id <> 51
WHERE
t1.campaign_id = 51;
Trying to remove duplicate based on three column like query will not find duplicate based on single column it will concatenate three column then remove duplicate rows based on the merge column.
I would appreciate if someone can share an easiest way of achieving this. I know this is not an appropraiote way but tried and its not working
Select concat(col1, col2, col3,) as newCol distinct newcCol from Table2
I know how to remove the table duplicate based on multiple columns using Excel VBA but do not know how to achieve this using mysql
Sub DelDupl()
Range("A1").CurrentRegion.RemoveDuplicates Columns:=Array(1, 2, 3), Header:=xlYes
End Sub
Table name is Table2 in Mysql
enter image description here
Sample Data
CREATE TABLE Table2(
col1 INT,
col2 varchar(10),
col3 INT,
col4 varchar(10),
col5 varchar(10),
col6 varchar(10),
col7 varchar(10));
INSERT INTO Table2 (col1,col2,col3,col4,col5,col6,col7)
VALUES ('1','A','123456','data1','data1','data1','data1'),
('2','B','78910','data2','data2','data2','data2'),
('3','C','45698','data3','data3','data3','data3'),
('1','A','123456','data1','data1','data1','data1'),
('2','B','78910','data2','data2','data2','data2'),
('3','C','45698','data3','data3','data3','data3'),
('4','D','85969','data5','data5','data5','data5');
The problem is there is no way to establish which to keep so I suggest you set up a staging table with a compound key, load it , truncate your existing table and load it from the staging table
How about this:
SELECT *
FROM table2 AS t
GROUP BY t.col1, t.col2, t.col3;
GROUP BY is normally used for aggregating functions(count, sum, max, etc), but will do the job for your purpose.
Edit:
Ok, since you actually need to delete duplicates, thats a bit more complicated, but it's possible. First we need to somehow discriminate duplicate rows, so we will temporary add a primary key. Then execute delete statement, while joining the table on itself to find duplicated rows. And lastly drop the primary key column we added.
add primary key column, we need some discrimination for duplicates
ALTER TABLE `table2` ADD COLUMN id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY;
join table on itself to get duplicates and delete them
DELETE t1
FROM `table2` t1
INNER JOIN `table2` t2
ON t1.col1 = t2.col1 AND t1.col2 = t2.col2 AND t1.col3 = t2.col3
WHERE t1.id < t2.id;
drop the primary key column
ALTER TABLE `table2` DROP COLUMN id;
As a note, the join is done on the columns which define duplication(col1, col2, col3) in your case. You can execute all 3 queries at once, just be sure you really need this data gone.
I have two tables with same elements, the only differnce is there ids' which are primary key and auto increment.
Table1 | Table2
id1(PK)| id2(PK)
col1 | col1
col2 | col2
col3 | col3
I know some quick ways to do that like,
INSERT INTO table2 SELECT * FROM table1 where id1 = 2
while using such method the content of table2 has id2 = 2 as it copies all the fields directly to table2 from table1, to restric that,
I can also use a method
INSERT INTO table2(col1,col2,col3) SELECT col1,col2,col3 FROM table1 WHERE id1 = 2
such way is good for short tables, but I have lot of columns in my table.
I need a quick way to copy all the columns from table1 to table2 leaving the primary columns which is id2, as it is autoincremented.
Its like I want to copy a specified row from table1 to table2 with different id2(which will be generated as its autoincremented).
Are there any possibilities.
If you do not want to mention column names but want to copy all, then try copy all data into a temp table, then drop pk_id field, and then copy rest of the fields into desired table, lastly drop the temp table.
Refer to one of my answers to a similar queries:
Mysql: Copy row but with new id
We can use temporary table to buffer first from main table and use it to copy to main table again. We can drop the pk field from the temp table and copy all other to the second table.
With reference to the answer by Tim Ruehsen in a referred posting:
CREATE TEMPORARY TABLE tmp_table SELECT * from first_table WHERE ...;
ALTER TABLE tmp_table drop pk_id; # drop autoincrement field
# UPDATE tmp_table SET ...; # just needed to change other unique keys
INSERT INTO second_table SELECT 0, tmp_table.* FROM tmp_table;
DROP TABLE tmp_table;
Copy all rows from table1 except that have same id in table2
INSERT INTO table2(col1, col2, col3)
SELECT t1.col1, t1.col2, t1.col3
FROM table1 t1
WHERE t1.id1 not in (SELECT id2 FROM table2);
How can we copy data from one table into another table which doesn't exist in the first one. In table one of the column is primary key.
INSERT INTO table SELECT * FROM db2.table;
ERROR 1062 (23000): Duplicate entry '100001' for key 'id_UNIQUE'
You can use the predicate NOT IN to do so like this:
INSERT INTO table1
SELECT *
FROM db2.table2
WHERE table1ReferenceID NOT IN(SELECT id_UNIQUE FROM table1);
This will checks whether this table1ReferenceID found in the first table or nor. Therefore, the SELECT clause will select all the rows from the second tables except those that is already presented in the first table table1.
Note that: the column table1ReferenceID is the reference of the id_UNIQUE in the second table.
Other alternatives for this, is to LEFT JOIN as suggested by #HamletHakobyan's answer and NOT EXISTS.
See it in action
Try this:
INSERT INTO table
SELECT T1.*
FROM db2.table T1
LEFT JOIN table T2
ON T1.Id = T2.Id
WHERE T2.Id IS NULL;
WHERE [primary_key_table1] not IN (SELECT [primary_key_table2] FROM [table2])
so i suggest something like
INSERT INTO table (SELECT * FROM db2.table WHERE id NOT IN (SELECT id FROM table));
Sometimes if I want to quickly copy records from one table to another (that has the same structure) I use a query like this:
INSERT INTO table2 SELECT * FROM
table1 WHERE id = SOME_VALUE
How can I add a ON DUPLICATE KEY UPDATE to this statement? I tried this:
INSERT INTO SELECT * FROM table1 WHERE
id = 1 ON DUPLICATE KEY UPDATE SELECT
* FROM table1 WHERE id = 1
But I get an error. Is there away to accomplish the query above with out individually listing each column in the query?
P.S. Yes, I realize that it is not good practice to have multiple tables with identical structures, but sometimes you just don't get control over everything in the workplace!
The below UPDATES if there is no PK duplication and INSERTs is there is:
REPLACE INTO table2(field1, field2, field3)
SELECT field1, field2,field3 FROM table1
WHERE id=1;
http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html Just use the SELECT field_name from the other table like in dnagirls example