I'm trying to insert data in a table which has the following columns :
id, date, client, device, user
I have created a primary key on id and unique key on the combination of client, device, and user.
While inserting data I getting the following Error:-
Cause: java.sql.SQLException: Duplicate entry '200-217-Xiaomi-M200' for key 'uk_user_client_device'
I checked the data in the table using the following query using:-
SELECT user, client, device, COUNT(1) rowcount FROM mytable GROUP BY user, client, device HAVING COUNT(1) > 1;
This got an empty set in response so I am certain there are no duplicate keys in the table.
I also went through the logs and I found that the data was inserted in the table at the same time I got this error. So, the data was inserted yet I got the duplicate entry error.
I also confirmed that this doesn't happen always. Sometimes, the data is inserted without any issue and sometimes I get an error and the data is inserted anyway.
I've seen a few questions regarding this with no definitive answer. I'm unable to figure out why this error is being thrown.
Of course this query returns no rows:
SELECT user, client, device, COUNT(1) as rowcount
FROM mytable
GROUP BY user, client, device
HAVING COUNT(1) > 1;
You have specified that client/device/user is a primary key (or at least unique. Hence, there are no duplicates. The database enforces this.
If you attempt to insert a row that would create a duplicate, the database returns an error. The row is not inserted. The database ensure the data integrity. Yay!
Your error is saying that the data in the insert either duplicates existing data in the table. Or, if you are inserting multiple rows, then the data has duplicates within the insert.
You can solve this in multiple ways. A typical method is to ignore the insert using on duplicate key update:
insert into mytable ( . . . )
. . .
on duplicate key update user = values(user); -- this is a no-op
The database prevents duplicates from being inserted. This construct prevents an error -- although no duplicate rows are inserted, of course.
Related
I've been searching the internet for a couple hours now and I'm not sure how to resolve this at all. So brief description is a customer posts orders to our system and they can supply a Customer Reference that our system will reject if that Customer Reference already exists.
I can't make the column in MySQL UNIQUE as different clients sometimes use the same Customer Reference and we do not require the Customer Reference so sometimes it's just left blank.
Originally I was just checking if the Customer Reference existed if necessary and then inserting the row if it did not exist. This works on 99.99% of cases, but I have a client that mass sends orders and those sometimes have duplicates. Which since they're posting quickly the select can happen before the first insert and duplicates arise.
I've switched to code like this below:(Shortened for example, this only runs if customerReference is not blank)
INSERT INTO ordersTable (clientID,customerReference,deliveryName) SELECT clientID, customerReference,deliveryName
FROM (SELECT 'clientID' as clientID, 'customerReference' as customerReference, 'deliveryName' as deliveryName) t
WHERE NOT EXISTS (SELECT 1 FROM ordersTable u WHERE u.customerReference = t.customerReference AND u.clientID = t.clientID);
This ends in deadlocks for any processes after the original row is inserted. I was hoping to avoid deadlocks?
My options it seems are:
Live with it deadlocking because I know if it deadlocks then the row already exists and instead of looking at affected_rows ==0 make it affected_rows <= 0.
Try to come up with some column that will make a unique record hash per order based on client ID and Customer Reference? and then do an "INSERT IGNORE" for that column?
I wasn't too confident in either solution so I thought it couldn't hurt to ask for advice first.
Have you tried using a transaction with a unique constraint on the uniqueID and clientID columns? This will prevent duplicates from being inserted, and you can catch the exception that is thrown when a replication is attempted to be inserted and handle it as needed.
INSERT INTO ordersTable (clientID,uniqueID,deliveryName)
VALUES ('clientID', 'uniqueID', 'deliveryName')
ON DUPLICATE KEY UPDATE deliveryName = VALUES(deliveryName);
Ok, you can also use "INSERT IGNORE" statement. This statement tells the server to insert the new record, but if there is a violation of a UNIQUE index or PRIMARY KEY, ignore the error and don't insert the new record.
INSERT IGNORE INTO ordersTable (clientID,uniqueID,deliveryName)
VALUES ('clientID', 'uniqueID', 'deliveryName');
I have some CRM data that exists in a MS SQL server, that I must move to mysql daily. I've got some python-pandas, read_sql() and to_sql() scripts that move the tables. I'm running into duplicate primary keys errors after doing some upsert logic. I have the GUID from CRM as the primary key for the table - in MySQL it is a varchar(64) datatype. Unsure what's triggering the duplicate warning.
mysql_table:
GUID-PK Name favorite_number modifiedon
00000B9D... Ben 10 '2017-01-01'
000A82A5... Harry 9 '2017-05-15'
000A9896... Fred 5 '2017-12-19'
(the GUIDs are longer, i'm shortening for the example)
I pull all the new records from MS SQL into a temporary table in MySQL based on modified dates that are greater than my current table. Some of these could be new records some could be records that already exist in my current table but have been updated.
mysql_temp_table:
GUID-PK Name favorite_number modifiedon
00000B9D... Ben 15 '2018-01-01'
000A82BB... John 3 '2018-03-15'
000A4455... Ray 13 '2018-04-01'
I want to replace any modified records, straight up, so I delete all the common records from the mysql_table. In this example, I want to remove Ben from the mysql_table, so that it can be replaced by Ben from the mysql_temp_table:
DELETE FROM mysql_table WHERE GUID-PK IN (SELECT GUID-PK FROM mysql_temp_table)
Then I want to just move the whole temp table into the replicated table with:
INSERT INTO mysql_table (SELECT * FROM temp_table)
But that gives me an error:
"Duplicate entry '0' for key 'PRIMARY'") [SQL: 'INSERT INTO mysql_table SELECT * FROM mysql_temp_table'
I can see that many of the GUID's start with '000', it seems like this is being interpreted as '0'. Shouldn't this be caught in the Delete-IN statement from above. i'm stuck on where to go next. Thanks in advance.
I suspect that the DELETE statement operation is failing with an error.
That's because the dash character isn't a valid character in an identifier. If the column name is really GUID-PK, then that needs to be properly escaped in the SQL text, either by enclosing it in backticks (the normal pattern in MySQL), or if sql_mode includes ANSI_QUOTES, then the identifiers can be enclosed in double quotes.
Another possibility is that temp_table does not have a PRIMARY or UNIQUE KEY constraint defined on the GUID-PK column, and there are multiple rows in temp_table that have the same value for GUID-PK, leading to a duplicate key exception on the INSERT into mysql_table.
Another guess (since we're not seeing the definition of the temp_table) is that the columns are in a different order, such that SELECT * FROM temp_table isn't returning columns in the order expected in mysql_table. I'd address that issue by explicitly listing the columns, of both the target table for the INSERT, and in the SELECT list.
Given that that GUID-PK column is a unique key, I would tend to avoid two separate statements (a DELETE followed by an INSERT), and just use INSERT ... ON DUPLICATE KEY UPDATE statement.
INSERT INTO mysql_table (`guid-pk`, `name`, `favorite_number`, `modifiedon` )
SELECT s.`guid-pk`, s.`name`, s.`favorite_number`, s.`modifiedon`
FROM temp_table s
ORDER
BY s.`guid-pk`
ON DUPLICATE KEY
UPDATE `name` = VALUES( `name` )
, `favorite_number` = VALUES( `favorite_number` )
, `modifiedon` = VALUES( `modifiedon` )
You may have AUTOCOMMIT disabled.
If you are performing both actions in the same TRANSACTION and do not have AUTOCOMMIT enabled your second READ COMMITTED statement will fail. INSERTS, UPDATES, and DELETES are executed using the READ COMMITTED Isolation Level
Your INSERT is being performed on the data set as it appeared before your DELETE. You need to either:
A. Explicitly COMMIT your DELETE within the TRANSACTION
or
B. Split the two statements into individual TRANSACTIONs
or
C. Renable AUTOCOMMIT
If this is not the case you will need to investigate your data sets for your DELETE and INSERT statements, because a DELETE will not just fail silently.
I have a CSV file which I need to enter into my Database. My modus operandi is a Bulk insert. One of the columns has a uniqueness constraint attached to it but it is not the primary column. If there is a duplicate entry, it correctly skips the line and does not enter it into the database. (On command line it indicates Duplicates: n, where n is the total number of duplicates).
Is there anyway I can retrieve the duplicate row numbers ? For instance, using Show Warnings or Show Errors, it states the last MySQL errors and warnings, is there anyway I can retrieve the duplicates from MySQL alone ?
Thanks.
You could enter the data into a temporary table first, without the uniqueness constraint, and perform a query to find all the duplicates.
SELECT unique_column, count(*) c
FROM temp_tablename
GROUP BY unique_column
HAVING c > 1;
Then copy it from the temporary table to the real table with:
INSERT IGNORE INTO tablename SELECT * FROM temp_tablename;
I have two tables, the first table has 400 rows. The second table holds the same records with the same count. Now the first table row count increases to 450. I want to insert only those 50 new rows into the second table. I don't need to update the first 400 records.
I am setting the unique index for the particular field (like empid). Now when I insert the first table data it returns the following error:
Duplicate entry 'xxxx' for key 'idx_confirm'
Please help me to fix this error.
Am using the below code to insert the record. But it allows duplicate entry..
insert ignore into tbl_emp_confirmation (fldemp_id,fldempname,fldjoindatefldstatus)
select fldempid, fldempname,DATE_FORMAT (fldjoindate,'%Y-%m-%d') as fldjoindate,fldstatus from tblempgeneral as n;
Modify your INSERT ... statement to INSERT IGNORE ....
See for example this post for an explanation.
You need to make sure that you have a unique index that prevents any duplicates, such as on the primary key.
How to get the primary key (assuming know his name by looking show keys) resulting from an insert into?
How to get the primary keys of rows affected by an update? (as in the previous case, independent of the key name).
How to get the primary keys returned from a select query (in the query even if the key is not one of the fields surveyed).
I need to SQLs commands I run after the inserts, updates and selects in my application to obtain such information, it is possible?
My database is MySQL.
I need only sqls because i am making a logic of cache queries to aplicate in many applications (java and php) and i wish that the logic be independent of language.
example:
select name from people
i need that a query executed after this return the pk of these people
SELECT LAST_INSERT_ID();
And seriously, putting "primary key from insert mysql" into Google gets you a Stack Overflow answer as the first result.
EDIT: more discussion based on comments.
If you want to see what rows are affected by an update, just do a SELECT with the same WHERE clause and JOIN criteria as the UPDATE statement, e.g.:
UPDATE foo SET a = 5 WHERE b > 10;
SELECT id FROM foo WHERE b > 10;
If you are INSERTing into a table that does not have an auto-increment primary key, you don't need to do anything special. You already know what the new primary key is, because you set it yourself in the INSERT statement. If you want code that can handle INSERT statements coming from outside of the code that will be tracking PK changes, then you'll either need to parse the INSERT statement, or have the calling code provide information about the primary key.