I'm running the following query to record aggregated stats for traffic according to the country of origin:
INSERT INTO stats_countries (country_id, country_traffic, country_updated)
VALUES (?,1,UNIX_TIMESTAMP())
ON DUPLICATE KEY UPDATE
country_traffic = country_traffic + 1,
country_updated = UNIX_TIMESTAMP()
Unfortunately it's running this query so frequently it seems to be causing some kind of row deadlock condition, where I have numerous connections going to sleep forever.
Is there any way I can fix this easily while keeping the "on duplicate" condition?
Or will I have to manually create one row for each country_id and just run a simple update?
Related
I've been searching the internet for a couple hours now and I'm not sure how to resolve this at all. So brief description is a customer posts orders to our system and they can supply a Customer Reference that our system will reject if that Customer Reference already exists.
I can't make the column in MySQL UNIQUE as different clients sometimes use the same Customer Reference and we do not require the Customer Reference so sometimes it's just left blank.
Originally I was just checking if the Customer Reference existed if necessary and then inserting the row if it did not exist. This works on 99.99% of cases, but I have a client that mass sends orders and those sometimes have duplicates. Which since they're posting quickly the select can happen before the first insert and duplicates arise.
I've switched to code like this below:(Shortened for example, this only runs if customerReference is not blank)
INSERT INTO ordersTable (clientID,customerReference,deliveryName) SELECT clientID, customerReference,deliveryName
FROM (SELECT 'clientID' as clientID, 'customerReference' as customerReference, 'deliveryName' as deliveryName) t
WHERE NOT EXISTS (SELECT 1 FROM ordersTable u WHERE u.customerReference = t.customerReference AND u.clientID = t.clientID);
This ends in deadlocks for any processes after the original row is inserted. I was hoping to avoid deadlocks?
My options it seems are:
Live with it deadlocking because I know if it deadlocks then the row already exists and instead of looking at affected_rows ==0 make it affected_rows <= 0.
Try to come up with some column that will make a unique record hash per order based on client ID and Customer Reference? and then do an "INSERT IGNORE" for that column?
I wasn't too confident in either solution so I thought it couldn't hurt to ask for advice first.
Have you tried using a transaction with a unique constraint on the uniqueID and clientID columns? This will prevent duplicates from being inserted, and you can catch the exception that is thrown when a replication is attempted to be inserted and handle it as needed.
INSERT INTO ordersTable (clientID,uniqueID,deliveryName)
VALUES ('clientID', 'uniqueID', 'deliveryName')
ON DUPLICATE KEY UPDATE deliveryName = VALUES(deliveryName);
Ok, you can also use "INSERT IGNORE" statement. This statement tells the server to insert the new record, but if there is a violation of a UNIQUE index or PRIMARY KEY, ignore the error and don't insert the new record.
INSERT IGNORE INTO ordersTable (clientID,uniqueID,deliveryName)
VALUES ('clientID', 'uniqueID', 'deliveryName');
I'm trying to insert data in a table which has the following columns :
id, date, client, device, user
I have created a primary key on id and unique key on the combination of client, device, and user.
While inserting data I getting the following Error:-
Cause: java.sql.SQLException: Duplicate entry '200-217-Xiaomi-M200' for key 'uk_user_client_device'
I checked the data in the table using the following query using:-
SELECT user, client, device, COUNT(1) rowcount FROM mytable GROUP BY user, client, device HAVING COUNT(1) > 1;
This got an empty set in response so I am certain there are no duplicate keys in the table.
I also went through the logs and I found that the data was inserted in the table at the same time I got this error. So, the data was inserted yet I got the duplicate entry error.
I also confirmed that this doesn't happen always. Sometimes, the data is inserted without any issue and sometimes I get an error and the data is inserted anyway.
I've seen a few questions regarding this with no definitive answer. I'm unable to figure out why this error is being thrown.
Of course this query returns no rows:
SELECT user, client, device, COUNT(1) as rowcount
FROM mytable
GROUP BY user, client, device
HAVING COUNT(1) > 1;
You have specified that client/device/user is a primary key (or at least unique. Hence, there are no duplicates. The database enforces this.
If you attempt to insert a row that would create a duplicate, the database returns an error. The row is not inserted. The database ensure the data integrity. Yay!
Your error is saying that the data in the insert either duplicates existing data in the table. Or, if you are inserting multiple rows, then the data has duplicates within the insert.
You can solve this in multiple ways. A typical method is to ignore the insert using on duplicate key update:
insert into mytable ( . . . )
. . .
on duplicate key update user = values(user); -- this is a no-op
The database prevents duplicates from being inserted. This construct prevents an error -- although no duplicate rows are inserted, of course.
I ve already seen some questions regarding this like below
MySQL “good” way to insert a row if not found, or update it if it is found
Now i have a summary table which gets updated with the qty every time say a sale occurs. so out of 1000 sales of an item only first time the insert executes and the rest of the times it would be update. My understanding is in Insert on Duplicate Key Update it tries to insert first and if it fails updates. so all 999 times the insert is not successfull
1) Is there a method to check Update first and if not updated then insert in a single statement?
2) which of the below methods would be desirable considering most of the cases update will be successfull
a) using Insert on Duplicate Key Update
b) Call Update; if no rows affected call insert
Right now i am using the second option(b). performance gain is very important here and also i m testing the first option. ill post the results here once done
INSERT ON DUPLICATE KEY UPDATE is the way to go. It does not perform a full insert or how you put it.
For this statement to work there has to be a primary key or unique key on the table and the corresponding columns have to be involved in the statement.
Before it's decided whether an insert or an update statement has to be done, the said keys are checked. This is usually really fast.
With your "update first" approach you gain nothing, no it gets even worse. The primary key lookup has to be done anyway. In the worst case you wasted time by having to look up the primary key two times. First for the update statement (which may not be necessary), then for the insert statement.
Have run into this weird problem where a simple query fails due to a deadlock
Here is the query
UPDATE myprobelmatictable SET mycolumn = (mycolum-0) WHERE id = '59'
The weird issue is that this query fails only when my php server is located on a slower server on a remote network
Before I run this query, the following happens
transaction starts
insert new row in table 5
select 1 row from myproblematictable
insert new row in table 6
update table 4
UPDATE myprobelmatictable SET mycolumn = (mycolum-0) WHERE id = '<id>'
update table 3
Commit Transaction
The strange thing is that the same query fails each time with the following error
Error Number: 1213</p><p>Deadlock found when trying to get lock; try restarting transaction
The innodb status command does not seem to mention myproblematictable
any clues?
This can be the result of another query updating the tables in a different order. I would try to see if there's a pre-determined order in which the tables should be updated, and if so, rewriting the order of the updates.
If not, I would suggest trying to find the offending quer(ies), and seeing what order they are updating the tables. What type of table engine are you using? Keep in mind that MyISAM locks the entire table.
I need a little help with SELECT FOR UPDATE (resp. LOCK IN SHARE MODE).
I have a table with around 400 000 records and I need to run two different processing functions on each row.
The table structure is appropriately this:
data (
`id`,
`mtime`, -- When was data1 set last
`data1`,
`data2` DEFAULT NULL,
`priority1`,
`priority2`,
PRIMARY KEY `id`,
INDEX (`mtime`),
FOREIGN KEY ON `data2`
)
Functions are a little different:
first function - has to run in loop on all records (is pretty fast), should select records based on priority1; sets data1 and mtime
second function - has to run only once on each records (is pretty slow), should select records based on priority2; sets data1 and mtime
They shouldn't modify the same row at the same time, but the select may return one row in both of them (priority1 and priority2 have different values) and it's okay for transaction to wait if that's the case (and I'd expect that this would be the only case when it'll block).
I'm selecting data based on following queries:
-- For the first function - not processed first, then the oldest,
-- the same age goes based on priority
SELECT id FROM data ORDER BY mtime IS NULL DESC, mtime, priority1 LIMIT 250 FOR UPDATE;
-- For the second function - only processed not processed order by priority
SELECT if FROM data ORDER BY priority2 WHERE data2 IS NULL LIMIT 50 FOR UPDATE;
But what I am experiencing is that every time only one query returns at the time.
So my questions are:
Is it possible to acquire two separate locks in two separate transactions on separate bunch of rows (in the same table)?
Do I have that many collisions between first and second query (I have troubles debugging that, any hint on how to debug SELECT ... FROM (SELECT ...) WHERE ... IN (SELECT) would be appreciated )?
Can ORDER BY ... LIMIT ... cause any issues?
Can indexes and keys cause any issues?
Key things to check for before getting much further:
Ensure the table engine is InnoDB, otherwise "for update" isn't going to lock the row, as there will be no transactions.
Make sure you're using the "for update" feature correctly. If you select something for update, it's locked to that transaction. While other transactions may be able to read the row, it can't be selected for update, updated or deleted by any other transaction until the lock is released by the original locking transaction.
To keep things clean, try explicitly starting a transaction using "START TRANSACTION", run your select "for update", do whatever you're going to do to the records that are returned, and finish up by explicitly executing a "COMMIT" to close out the transaction.
Order and limit will have no impact on the issue you're experiencing as far as I can tell, whatever was going to be returned by the Select will be the rows that get locked.
To answer your questions:
Is it possible to acquire two separate locks in two separate transactions on separate bunch of rows (in the same table)?
Yes, but not on the same rows. Locks can only exist at the row level in one transaction at a time.
Do I have that many collisions between first and second query (I have troubles debugging that, any hint on how to debug SELECT ... FROM (SELECT ...) WHERE ... IN (SELECT) would be appreciated )?
There could be a short period where the row lock is being calculated, which will delay the second query, however unless you're running many hundreds of these select for updates at once, it shouldn't cause you any significant or noticable delays.
Can ORDER BY ... LIMIT ... cause any issues?
Not in my experience. They should work just as they always would on a normal select statement.
Can indexes and keys cause any issues?
Indexes should exist as always to ensure sufficient performance, but they shouldn't cause any issues with obtaining a lock.
All points in accepted answer seem fine except below 2 points:
"whatever was going to be returned by the Select will be the rows that get locked." &
"Can indexes and keys cause any issues?
but they shouldn't cause any issues with obtaining a lock."
Instead all the rows which are internally read by DB during deciding which rows to select and return will be locked. For example below query will lock all rows of the table but might select and return only few rows:
select * from table where non_primary_non_indexed_column = ? for update
Since there is no index, DB will have to read the entire table to search for your desired row and hence lock entire table.
If you want to lock only one row either you need to specify its primary key or an indexed column in the where clause. Thus indexing becomes very important in case of locking only the appropriate rows.
This is a good reference - https://dev.mysql.com/doc/refman/5.7/en/innodb-locking-reads.html