I have an issue with duplicate entries in a database and due to the nature of the problem the easiest way to fix it would be to remove current duplicate rows and prevent further duplicates from being added .
Here is the table structure :
| a | b | c |
user url1 token1
photo url1 token2
action action1 token3
user url1 token4
photo url1 token5
action action2 token6
I want to prevent duplicate entries only when 2 columns are duplicated, in this case a and b .
So here we have user | url1 and photo | url1 duplicated twice.
I want to prevent any further duplicates from being added when both columns match another row at same time but the queries I found so far will consider each column separately and preventing any further duplicates to be added to any of them .
Can I achieve this with a mysql query using unique index ?
I tried using the following code :
Using ALTER TABLE `targets` ADD UNIQUE (
`a` ,
`b`
);
Your question:"Can I achieve this with a mysql query using unique index ?"
Answer is 100% yes.
There are two ways of creating index:
1. CREATE UNIQUE INDEX index_name
ON table_name (column1, column2, ...);
2. ALTER TABLE table_name
ADD UNIQUE index_name (column1, column2, ...);
However, this will only work if your table doesn't have existing duplicate data. Otherwise you'll receive an error message like this:
Query: CREATE UNIQUE INDEX index_name ON targets (a, b)
Error Code: 1062
Duplicate entry 'photo-url1' for key 'index_name'
Therefore, you need to:
create a new empty table similar to your targets table.
create unique index.
INSERT IGNORE data from the old table.
Rename targets to targets_old and targets_new to targets.
Example:
CREATE TABLE targets_new LIKE targets;
CREATE UNIQUE INDEX index_name
ON targets_new (a, b);
INSERT IGNORE INTO targets_new SELECT * FROM targets;
RENAME TABLE targets TO targets_old;
RENAME TABLE targets_new TO targets;
Thanks for the replies guys, but I found the solution in the meantime and it was much simpler !
It's called unique composite key and it allows to do exactly what I wanted :
ALTER TABLE targets ADD UNIQUE KEY `uidx` (a, b, c);
Problem fixed :)
i don't think you can specify a unique doublet property.
edit : mention by Salmon : An index specification of the form (key_part1, key_part2, ...) creates an index with multiple key parts
https://dev.mysql.com/doc/refman/8.0/en/create-index.html#create-index-unique
you could try :
SELECT * FROM database1.table1 WHERE a='user' AND b='url1';
if got return a number of rows then don't add to the database.
this way you can fully control what goes into the database instead of letting the table automatically ignore the fail condition.
Related
I get database table which contain postal numbers and regions for my country. That table have all information but i need to change it for my purpose.
I need to eliminate all rows that have duplicate content in specific column.
Check screenshot to see result
I want to remove all duplicate rows which have postanski_broj (postal_number) the some. That number need to be unique. I try manualy to set that column to unique but i get duplicate entry when i try to execute statment.
ID is primary key with auto increment.
postanski_broj column is VARCHAR which represent postal_code
naselje column is VARCHAR which represent region
One region can have one postal_code
I try
ALTER TABLE poste ADD UNIQUE INDEX idx_postanski_br (postanski_broj);
00:03:20 ALTER TABLE poste ADD UNIQUE INDEX idx_postanski_br
(postanski_broj) Error Code: 1062. Duplicate entry '11158' for key
'idx_postanski_br' 0.118 sec
ALTER IGNORE TABLE poste ADD UNIQUE INDEX idx_postanski_br (postanski_broj);
00:04:17 ALTER IGNORE TABLE poste ADD UNIQUE INDEX idx_postanski_br
(postanski_broj) Error Code: 1064. You have an error in your SQL
syntax; check the manual that corresponds to your MySQL server version
for the right syntax to use near 'IGNORE TABLE poste ADD UNIQUE INDEX
idx_postanski_br (postanski_broj)' at line 1 0.00037 sec
Anyone have sugestion? Thanks
If you have other columns with different values than the ones you've shown there (except for id), deleting should be your last choice.
I usually would duplicate the table first:
CREATE TABLE poste_new LIKE poste;
add unique index to the newly created poste_new table:
ALTER TABLE poste_new ADD UNIQUE INDEX idx_postanski_br (postanski_broj);
insert the data from poste into poste_new with IGNORE option to skip duplicates based on the unique index:
INSERT IGNORE INTO poste_new SELECT * FROM poste;
rename the tables:
RENAME TABLE poste TO poste_old;
RENAME TABLE poste_new TO poste;
The good thing about this is that you've minimized the risk of wrong delete and if you're not satisfied with the new table, you still have the old table intact - effectively making it a backup.
This solution can take too much time for big tables. Best way of solving this is: Remove duplicate rows in MySQL
You have to delete the rows before applying the unique constraint. Be careful applying this:
DELETE p1 FROM poste p1
INNER JOIN poste p2
WHERE
p1.id < p2.id AND
p1.postanski_broj = p2.postanski_broj;
This should remove the duplicated ones and will keep only the ones with the higher id (id=168044 in your example).
I have table with 7 columns where I want to combine three column and want to remove duplicates. Your help will be appreciated.
I'm guessing you want to update the table SERIAL_NUMBERS with just a single row of combined warranty_indicator, account, date_sold data. If that's true then I will suggest the following.
Duplicate SERIAL_NUMBERS table:
CREATE TABLE SERIAL_NUMBERS_NEW LIKE SERIAL_NUMBERS;
Add unique constraint - combination of warranty_indicator, account, date_sold:
ALTER TABLE SERIAL_NUMBERS_NEW
ADD CONSTRAINT Index1 UNIQUE (warranty_indicator,account,date_sold);
Insert data from SERIAL_NUMBERS table to SERIAL_NUMBERS_NEW using INSERT IGNORE ; to ignore duplicates:
INSERT IGNORE INTO SERIAL_NUMBERS_NEW
SELECT * FROM SERIAL_NUMBERS;
Rename old and new table:
RENAME TABLE SERIAL_NUMBERS TO SERIAL_NUMBERS_OLD;
RENAME TABLE SERIAL_NUMBERS_NEW TO SERIAL_NUMBERS;
Check data:
SELECT * FROM SERIAL_NUMBERS_OLD;
SELECT * FROM SERIAL_NUMBERS;
Keep in mind that any future data inserted will treat duplicates according to the unique constraint. Therefore, if you have program running the INSERT syntax, make sure you update it to INSERT IGNORE.
Demo fiddle
This seems like a simple task, but I am struggling to find a way to do this. I have two tables (A and B) with the same structure. Both have an auto-increment primary key. I want the data from table B to go in to table A. I tried:
insert into A select * from B
However I get an error that a primary key already exists. I would like the new rows from table B to get a new primary key when I insert them in to table A - so basically discard the primary key from B but insert all the other columns. Seems like it should be a simple query but I can't figure it out. Thank you.
You're also selecting (and inserting) the auto_increment id, which fails if such id already exists. To let mysql automatically assign id just select and insert all values besides the id:
INSERT INTO A (foo, bar, baz)
SELECT foo, bar, baz FROM B
You must specify the columns you are want to insert from table B to table A, since you want to create a new id you can use a query as follow assuming column1 is they key that we don't want to insert.
insert into A (column2, column3) select column2, column3 from B
You can also lookup at this link that has some good query examples.
I have a huge table of products but there are lot of duplicate entries. The table has more than10 Thousand entries and I want to remove the duplicate entries in it without manually finding and deleting it. Please let me know if you can provide me a solution for this
You could use SELECT DISTINCT INTO TempTable, drop the original table, and then rename the temp one.
You should also add primary and unique keys to avoid this sort of thing in the future.
for full row duplicates try this.
select distinct * into mytable_tmp from mytable
drop table mytable
alter table mytable_tmp rename mytable
Seems the below statements will help you in resolving your requirements.
if the table(foo) has primary key field
First step
store key values in temporary table, give your unique conditions in group by clause
if you want to delete the duplicate email id, give email id in group by clause and give the primary key name in
select clause like either min(primarykey) or max(primarykey)
CREATE TEMPORARY TABLE temptable AS SELECT min( primarykey ) FROM foo GROUP BY uniquefields;
Second step
call the below delete statement and give the table name and primarykey columns
DELETE FROM foo WHERE primarykey NOT IN (SELECT * FROM temptable );
execute both the query combined in your query analyser or db tool.
If the table(foo) doesn't have a primary key filed
step 1
CREATE TABLE temp_table AS SELECT * FROM foo GROUP BY field or fileds;
step 2
DELETE FROM foo;
step 3
INSERT INTO foo select * from temp_table;
There are different solutions to remove duplicate rows and it fully depends upon your scenario to make use of one from them. The simplest method is to alter the table making the Unique Index on Product Name field:
alter ignore table products add unique index `unique_index` (product_name);
You can remove the index after getting all the duplicate rows deleted:
alter table products drop index `unique_index`;
Please let me know if this resolves the issue. If not I can give you alternate solutions for that.
You can add more than one column to a group by. I.E.
SELECT * from tableName GROUP BY prod_name HAVING count(prod_name) > 1
That will show the unique products. You can write it dump it to new table and drop the existing one.
I have a category say ecommerce.add ebay and amazon.when i update ebay as amazon,it should n't update.How do i d it?
I suggest you check out unique indexes and primary keys. These will cause an insert or update to fail rather that allow duplicate entries to be made.
CREATE UNIQUE INDEX name_unique ON tablename (name(10));
Replace name_unique with the name you want for the index, tablename with the name of your table, and name(10) with the column name and how many characters you want to be unique (the length of the column if you want the entire value to be unique).