Isolation level is READ COMMITTED
The one query is looking for records where service_id is 0 ... the other is looking where service_id is NOT IN (0, ... others ... );
I'd think they would be locking different rows?
------------------------
LATEST DETECTED DEADLOCK
------------------------
2017-08-18 09:01:24 7f2d05641700
*** (1) TRANSACTION:
TRANSACTION 201694975, ACTIVE 1 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 46 lock struct(s), heap size 6544, 194 row lock(s)
MySQL thread id 33600289, OS thread handle 0x7f2d0812b700, query id 3703173090 inf-rtpctllb02-prd.rtp.netapp.com 10.60.56.150 ctl Copying to tmp table
SELECT
re.*,
r.config_id,
r.reserve_all_or_nothing,
r.owner,
r.charges
FROM
`job_charge` AS re,
`job` AS r WHERE
re.job_id = r.id AND ((re.status ='dispatched') or (re.status= 'running') or (re.status= 'held') or (re.status= 'reserved')) AND ((re.service_id ='0')) AND r.disable = 0 ORDER BY r.priority,r.id LIMIT 10000 FOR UPDATE
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1485 page no 987 n bits 104 index `PRIMARY` of table `ctl`.`job_charge` trx table locks 2 total table locks 2 trx id 201694975 lock_mode X locks rec but not gap waiting lock hold time 1 wait time before grant 0
*** (2) TRANSACTION:
TRANSACTION 201691925, ACTIVE 185 sec fetching rows
mysql tables in use 4, locked 2
1164 lock struct(s), heap size 128552, 2 row lock(s)
MySQL thread id 33599597, OS thread handle 0x7f2d05641700, query id 3703158120 inf-rtpctllb02-prd.rtp.netapp.com 10.60.56.150 ctl updating
UPDATE
`job_charge`
SET
service_id = '0'
WHERE
service_id NOT IN ('0','ctl5-staging_command-launcher.674d8c96-7c76-11e7-bc6c-ee0cf095fd00','inf-mesos-slave001.ctl.gdl.englab.netapp.com:mesos-6b256982-4ef1-4a84-ba60-58245ee7406d-S63.3987fd54-ee31-4c81-add4-4be53a6ed363:80','ctl5-staging_scheduler.912d008f-7c76-11e7-bc6c-ee0cf095fd00','ctl5-production_capacity-manager.6a869ee7-7919-11e7-bc6c-ee0cf095fd00','ctl5-production_scheduler.91de7d76-7919-11e7-bc6c-ee0cf095fd00','mysql','inf-mesos-slave001.ctl.gdl.englab.netapp.com:mesos-6b256982-4ef1-4a84-ba60-58245ee7406d-S63.48fe0555-83e9-4811-bcbc-f301da498fa6:80','ctl5-production_cleaner.6a86c5fa-7919-11e7-bc6c-ee0cf095fd00','ctl5-production_command-launcher.9f97a534-8413-11e7-bc6c-ee0cf095fd00','ctl5-production_reservation-manager.7ac1771d-7a9e-11e7-bc6c-ee0cf095fd00','ctl5-s
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1485 page no 987 n bits 104 index `PRIMARY` of table `ctl`.`job_charge` trx table locks 1 total table locks 2 trx id 201691925 lock_mode X locks rec but not gap lock hold time 13 wait time before grant 12
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1485 page no 1606 n bits 88 index `PRIMARY` of table `ctl`.`job_charge` trx table locks 1 total table locks 2 trx id 201691925 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0
*** WE ROLL BACK TRANSACTION (1)
Yes. Both must be different rows. You can see the page no 987 and 1606 as indicated in the statement
Here Transaction 1 is
SELECT
re.*,
r.config_id,
r.reserve_all_or_nothing,
r.owner,
r.charges
FROM
`job_charge` AS re,
`job` AS r WHERE
re.job_id = r.id AND ((re.status ='dispatched') or (re.status= 'running') or (re.status= 'held') or (re.status= 'reserved')) AND ((re.service_id ='0')) AND r.disable = 0 ORDER BY r.priority,r.id LIMIT 10000 FOR UPDATE
Transaction 2 is
UPDATE
`job_charge`
SET
service_id = '0'
WHERE
service_id NOT IN ('0','ctl5-staging_command-launcher.674d8c96-7c76-11e7-bc6c-ee0cf095fd00','inf-mesos-slave001.ctl.gdl.englab.netapp.com:mesos-6b256982-4ef1-4a84-ba60-58245ee7406d-S63.3987fd54-ee31-4c81-add4-4be53a6ed363:80','ctl5-staging_scheduler.912d008f-7c76-11e7-bc6c-ee0cf095fd00','ctl5-production_capacity-manager.6a869ee7-7919-11e7-bc6c-ee0cf095fd00','ctl5-production_scheduler.91de7d76-7919-11e7-bc6c-ee0cf095fd00','mysql','inf-mesos-slave001.ctl.gdl.englab.netapp.com:mesos-6b256982-4ef1-4a84-ba60-58245ee7406d-S63.48fe0555-83e9-4811-bcbc-f301da498fa6:80','ctl5-production_cleaner.6a86c5fa-7919-11e7-bc6c-ee0cf095fd00','ctl5-production_command-launcher.9f97a534-8413-11e7-bc6c-ee0cf095fd00','ctl5-production_reservation-manager.7ac1771d-7a9e-11e7-bc6c-ee0cf095fd00','ctl5-s
From the given message, we can see that transaction 1 is waiting for an Exclusive Lock (Denoted by X lock, lock needed for WRITING some value into the table) on the table 'ctl' primary key.
But at the same time, transaction 2 came into the picture which is already holding an X lock on the 'ctl' table primary key (page no 987). So, as transaction 2 is already having an X lock on 'ctl', transaction 1 can't get X lock and hence it is waiting.
But transaction 2 itself is waiting for another X lock on 'ctl' (different row from above, page no 1606). I think this row is being held by transaction 1.
So,
transaction 1 is holding a lock on a row in page no 1606 for which transaction 2 is waiting
and
transaction 2 is holding a lock on row in page no 987 for which transaction 1 is waiting
So, both are waiting for each other and hence deadlock occurred.
Related
Consider the following schema in mysql:
create table foo(
id int not null primary key auto_increment,
name varchar(32) not null,
unique key(name)
);
And there is a record with name "abc" in the table.
I have a transaction (RC):
start transaction;
delete from foo where name = "abc";
insert into foo(name) values("abc");
commit;
If there are two concurrent transactions, the dead lock will happen.
| TX A | TX B
---------------------------------------------------------------------
Step 1 | start transaction; |
| delete name="abc"; |
---------------------------------------------------------------------
Step 2 | | start transaction;
| | delete name="abc";
| | <wait for lock>
---------------------------------------------------------------------
Step 3 | insert name="abc"; | <deadlock detected, exit>
---------------------------------------------------------------------
Step 4 | commit; |
---------------------------------------------------------------------
I'm wondering why this sequence causes the deadlock.
In the mysql doc says (https://dev.mysql.com/doc/refman/8.0/en/innodb-locks-set.html)
If a duplicate-key error occurs, a shared lock on the duplicate index
record is set. This use of a shared lock can result in deadlock should
there be multiple sessions trying to insert the same row if another
session already has an exclusive lock. This can occur if another session
deletes the row.
I suppose when transaction A runs the "delete" statement, it has acquired the X lock of the record "abc". When the "insert" statement executes, it tries to acquire the S lock due to the "duplicate key error". Shouldn't it get the S lock since it has got the X lock of the same record? Why deadlock happens here?
I reproduced the deadlock, and got the innoDB status log as follow:
------------------------
LATEST DETECTED DEADLOCK
------------------------
2019-10-18 18:35:14 0x7f1dfc738700
*** (1) TRANSACTION:
TRANSACTION 26547965, ACTIVE 6 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 1136, 1 row lock(s)
/* ApplicationName=DataGrip 2019.1.1 */ delete from foo where name='abc'
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 3011 page no 4 n bits 224 index IDX_NAME of table `foo` trx id 26547965 lock_mode X locks rec but not gap waiting
Record lock, heap no 153 PHYSICAL RECORD: n_fields 2; ....
*** (2) TRANSACTION:
TRANSACTION 26547960, ACTIVE 10 sec inserting
mysql tables in use 1, locked 1
4 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 2
/* ApplicationName=DataGrip 2019.1.1 */ INSERT INTO foo(id, name)
VALUES (1, 'abc')
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 3011 page no 4 n bits 224 index IDX_NAME of table `foo` trx id 26547960 lock_mode X locks rec but not gap
Record lock, heap no 153 PHYSICAL RECORD: ...
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 3011 page no 4 n bits 224 index IDX_NAME of table `foo` trx id 26547960 lock mode S waiting
Record lock, heap no 153 PHYSICAL RECORD: ....
*** WE ROLL BACK TRANSACTION (1)
The log explains the reason clearly, TX B waiting X lock held by TX A, at the same time, TX A waiting S lock which is block by TX B's lock request.
According to Mysql doc:
If a duplicate-key error occurs, a shared lock on the duplicate index record is set. This use of a shared lock can result in deadlock should there be multiple sessions trying to insert the same row if another session already has an exclusive lock. "
Insert statement does acquire S lock at some point, so the reason for why deadlock happens is very clear.
But problem is:
according to the mysql doc, insert statement will acquire a S lock if a duplicate-key error occurs, which is not happened in current case we discuss
why insert statement still acquire a S lock when current transaction already holds the X lock, X lock is enough for doing a current read to check the duplicate key error. so what does it use for ?
We are struggling with one deadlock which is occuring multiple times a day in our production environment.
------------------------
LATEST DETECTED DEADLOCK
------------------------
2018-12-27 19:07:34 7fcef1959700
*** (1) TRANSACTION:
TRANSACTION 2125001468, ACTIVE 2 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 1
MySQL thread id 42190185, OS thread handle 0x7fcffc0b1700, query id 918842488 --- updating
UPDATE synchronization SET service_synchronized_at = NULL WHERE id = 116212
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 615 page no 288 n bits 528 index `PRIMARY` of table `app`.`synchronization` trx table locks 1 total table locks 2 trx id 2125001468 lock_mode X locks rec but not gap waiting lock hold time 2 wait time before grant 0
*** (2) TRANSACTION:
TRANSACTION 2125001355, ACTIVE 5 sec fetching rows
mysql tables in use 2, locked 2
25216 lock struct(s), heap size 3683880, 5297668 row lock(s), undo log entries 94
MySQL thread id 42189517, OS thread handle 0x7fcef1959700, query id 918842042 --- updating
UPDATE synchronization s SET s.service_synchronized_at = now() WHERE s.service_synchronized_at IS NULL AND s.user_id IN (* time consuming select to determine which users should be updated *)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 615 page no 288 n bits 528 index `PRIMARY` of table `app`.`synchronization` trx table locks 2 total table locks 2 trx id 2125001355 lock_mode X lock hold time 3 wait time before grant 0
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 615 page no 2313 n bits 472 index `PRIMARY` of table `app`.`synchronization` trx table locks 2 total table locks 2 trx id 2125001355 lock_mode X waiting lock hold time 0 wait time before grant 0
*** WE ROLL BACK TRANSACTION (1)
In one query I want to set service_synchronized_at = NULL. service_synchronized_at is always not null in first query before update.
In second query I add a where condition s.service_synchronized_at IS NULL thinking it would result in non-locking rows with not null values. Guess I was wrong.
The table has only primary index on id and unique constraint on user_id (and of course foreign key on user_id).
Any help is welcome.
I have been getting deadlocks while creating orders in a transaction.
Spree Adjustments table acquires a lock and keeps on updating Promotions or TaxOns. In the meanwhile, other request or order needs to update the same records.
------------------------
LATEST DETECTED DEADLOCK
------------------------
2016-09-17 20:40:15 7fee358d0b00
*** (1) TRANSACTION:
TRANSACTION 3880159183, ACTIVE 2 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 10 lock struct(s), heap size 2936, 7 row lock(s), undo log entries 18
MySQL thread id 128414, OS thread handle 0x7fee6fd6ab00, query id 53800114 172.31.34.254 connect_api_user updating
UPDATE `my_variants` SET `my_variants`.`quantity` = 2, `my_variants`.`updated_at` = '2016-09-17 20:40:43' WHERE `my_variants`.`id` = 108430
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2414 page no 1809 n bits 88 index `PRIMARY` of table `marketplace`.`my_variants` trx table locks 5 total table locks 2 trx id 3880159183 lock_mode X locks rec but not gap waiting lock hold time 2 wait time before grant 0
*** (2) TRANSACTION:
TRANSACTION 3880159065, ACTIVE 3 sec starting index read
mysql tables in use 1, locked 1
22 lock struct(s), heap size 2936, 15 row lock(s), undo log entries 36
MySQL thread id 125879, OS thread handle 0x7fee358d0b00, query id 53804455 172.31.34.254 connect_api_user updating
UPDATE `spree_adjustments` SET `spree_adjustments`.`eligible` = 0 WHERE `spree_adjustments`.`adjustable_id` = 2298885 AND `spree_adjustments`.`adjustable_type` = 'Spree::LineItem' AND `spree_adjustments`.`source_type` = 'Spree::PromotionAction' AND (`spree_adjustments`.`id` != 665173)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 2414 page no 1809 n bits 88 index `PRIMARY` of table `marketplace`.`teni_variants` trx table locks 13 total table locks 2 trx id 3880159065 lock_mode X locks rec but not gap lock hold time 2 wait time before grant 0
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1544 page no 2024 n bits 432 index `index_spree_adjustments_on_adjustable_id_and_adjustable_type` of table `marketplace`.`spree_adjustments` trx table locks 13 total table locks 2 trx id 3880159065 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0
*** WE ROLL BACK TRANSACTION (1)
I have a running system with a MySQL database engine.
Running the SHOW ENGINE INNODB STATUS command shows the:
------------------------
LATEST DETECTED DEADLOCK
------------------------
*** (1) TRANSACTION:
UPDATE db.alarm
LEFT JOIN (db.event, db.alarm1)
ON db.event.idevent = db.alarm.idevent
AND db.alarm1.idevent = db.alarm.idevent
SET
idalarmseverity = NAME_CONST('alarmseveritycleared',1),
lastmodifieddate = NAME_CONST('moddate',_binary'2015-01-07 09:02:00' COLLATE 'binary')
WHERE db.event.ideventsource = NAME_CONST('sourceid',3)
AND (db.alarm1.idorder = NAME_CONST('orderid',18894) OR NAME_CONST('orderid',18894) = 0)
AND db.alarm.idalarmseverity > 1
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 32942 n bits 440 index `GEN_CLUST_INDEX` of table `db`.`alarm` trx id 17 2553624600 lock_mode X locks rec but not gap waiting
*** (2) TRANSACTION:
UPDATE db.alarm
LEFT JOIN (db.event, db.alarm2)
ON db.event.idevent=db.alarm.idevent
AND db.alarm2.idevent=db.alarm.idevent
SET
idalarmseverity= NAME_CONST('alarmseveritycleared',1),
lastmodifieddate= NAME_CONST('moddate',_binary'2015-01-07 09:02:00' COLLATE 'binary')
WHERE db.event.ideventsource= NAME_CONST('sourceid',3)
AND db.alarm.idalarmseverity > 1
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 32942 n bits 440 index `GEN_CLUST_INDEX` of table `db`.`alarm` trx id 17 2553624599 lock_mode X locks rec but not gap
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 32941 n bits 440 index `GEN_CLUST_INDEX` of table `db`.`alarm` trx id 17 2553624599 lock_mode X locks rec but not gap waiting
At this point I don't know what the 2nd TRANSACTION is waiting for.
The 1st transaction is waiting for a lock (page no 32942) which is hold by the 2nd transaction. That's clear.
But the 2nd transaction is waiting for a lock (page no 32941) which is not hold by anyone.
Or if the 2 locks are the same (or overlaps) than it is the same transaction that currently holds the lock. I don't understand it well enough so it would be great if someone could explain to me how this deadlock could happen and how the UPDATE should be modified to prevent future deadlocks like this.
Thanks
This InnoDB deadlock is really making me pull my hair. As far as I can see:
the transaction (1) is waiting for the PRIMARY on "applications"
The latter has been acquired by (2) for some rather long running updates (SELECT * FROM applications WHERE ID = xxxx FOR UPDATE)
So far, so good - one would expect (1) to wait for the lock and then get on with its work.
However, once (2) gets ready to save it's work (and commit the transaction), it fails with a deadlock since for some reason (1) has managed to get a lock on some secondary index. How the hell did (1) manage to get any locks on the row if the PRIMARY is being held by (2).
One would expect that if (2) originally acquired the PRIMARY lock (SELECT * FROM applications WHERE ID = xxxx FOR UPDATE) it would also have set locks on all the secondary indexes. Is it possible that it will not lock the "tasked" index if tasked==NULL thus allowing (1) acquire a lock on "tasked" before even getting a lock on PRIMARY?
I have had no luck replication this scenario..
Thank you!
Lauri
------------------------
LATEST DETECTED DEADLOCK
------------------------
130428 17:04:06
*** (1) TRANSACTION:
TRANSACTION A369A8C, ACTIVE 1 sec fetching rows
mysql tables in use 3, locked 3
LOCK WAIT 217 lock struct(s), heap size 31160, 636 row lock(s)
MySQL thread id 13310554, OS thread handle 0x7f06cc2d7700, query id 177699568 217.146.78.151 shard67 Sending data
SELECT `applications`.* FROM `applications`
LEFT JOIN `applicants` ON applicants.ID = applications.applicant_ID
LEFT JOIN `regions` ON regions.ID = applicants.region_ID WHERE (status <> 'Blank') AND (status <> 'Closed') AND (revised < 1367154245) AND (tasked IS NULL OR tasked < 1367147045) AND (commence_year >= '2013') AND (regions.instance_ID = '1') ORDER BY `tasked` ASC, `ID` ASC LIMIT 20 FOR UPDATE
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 20021 page no 1192 n bits 80 index `PRIMARY` of table `dream-shard67`.`applications` trx id A369A8C lock_mode X locks rec but not gap waiting
*** (2) TRANSACTION:
TRANSACTION A369A87, ACTIVE 1 sec updating or deleting
mysql tables in use 1, locked 1
16 lock struct(s), heap size 3112, 22 row lock(s), undo log entries 5
MySQL thread id 13310563, OS thread handle 0x7f06cc151700, query id 177699599 217.146.76.127 shard67
UPDATE `applications` SET `revised` = '1367157846', `tasked` = '1367157846', `revision_ID` = '140649', `xml` = 'Zms6\noMmI$%[v....snipped binary data
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 20021 page no 1192 n bits 72 index `PRIMARY` of table `dream-shard67`.`applications` trx id A369A87 lock_mode X locks rec but not gap
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 20021 page no 292 n bits 1280 index `tasked` of table `dream-shard67`.`applications` trx id A369A87 lock_mode X locks rec but not gap waiting
*** WE ROLL BACK TRANSACTION (2)
Make sure your doing most of these:
http://dev.mysql.com/doc/refman/5.0/en/innodb-deadlocks.html