MySQL (InnoDB) deadlock - mysql

We are struggling with one deadlock which is occuring multiple times a day in our production environment.
------------------------
LATEST DETECTED DEADLOCK
------------------------
2018-12-27 19:07:34 7fcef1959700
*** (1) TRANSACTION:
TRANSACTION 2125001468, ACTIVE 2 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 1
MySQL thread id 42190185, OS thread handle 0x7fcffc0b1700, query id 918842488 --- updating
UPDATE synchronization SET service_synchronized_at = NULL WHERE id = 116212
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 615 page no 288 n bits 528 index `PRIMARY` of table `app`.`synchronization` trx table locks 1 total table locks 2 trx id 2125001468 lock_mode X locks rec but not gap waiting lock hold time 2 wait time before grant 0
*** (2) TRANSACTION:
TRANSACTION 2125001355, ACTIVE 5 sec fetching rows
mysql tables in use 2, locked 2
25216 lock struct(s), heap size 3683880, 5297668 row lock(s), undo log entries 94
MySQL thread id 42189517, OS thread handle 0x7fcef1959700, query id 918842042 --- updating
UPDATE synchronization s SET s.service_synchronized_at = now() WHERE s.service_synchronized_at IS NULL AND s.user_id IN (* time consuming select to determine which users should be updated *)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 615 page no 288 n bits 528 index `PRIMARY` of table `app`.`synchronization` trx table locks 2 total table locks 2 trx id 2125001355 lock_mode X lock hold time 3 wait time before grant 0
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 615 page no 2313 n bits 472 index `PRIMARY` of table `app`.`synchronization` trx table locks 2 total table locks 2 trx id 2125001355 lock_mode X waiting lock hold time 0 wait time before grant 0
*** WE ROLL BACK TRANSACTION (1)
In one query I want to set service_synchronized_at = NULL. service_synchronized_at is always not null in first query before update.
In second query I add a where condition s.service_synchronized_at IS NULL thinking it would result in non-locking rows with not null values. Guess I was wrong.
The table has only primary index on id and unique constraint on user_id (and of course foreign key on user_id).
Any help is welcome.

Related

mysql innodb_locks table dump to file

According to mysql reference about InnoDB Transaction and Locking Information page:
The data exposed by the transaction and locking tables (INNODB_TRX, INNODB_LOCKS, and INNODB_LOCK_WAITS) represents a glimpse into fast-changing data.
For performance reasons, and to minimize the chance of misleading joins between the transaction and locking tables, InnoDB collects the required transaction and locking information into an intermediate buffer whenever a SELECT on any of the tables is issued.
So I'd like to know if there is a way to write this info/buffer into a file?
I don't think there's any way to access the buffer you speak of directly. You can use SELECT queries against the named INFORMATION_SCHEMA tables, which will read from those buffers indirectly.
An alternative is to make the output of SHOW ENGINE INNODB STATUS be dumped to the MySQL Server error log every 15 seconds. You can optionally make this status include lock information. See https://dev.mysql.com/doc/refman/5.6/en/innodb-enabling-monitors.html
Example: I enabled the InnoDB lock monitor with lock options.
mysql> set global innodb_status_output_locks=on;
mysql> set global innodb_status_output=on;
Then I created one transaction, inserting into a test table. But don't commit yet.
mysql> BEGIN;
mysql> INSERT INTO t VALUES (1,1);
In a second window, I begin another transaction, with another insert designed to conflict with the first one.
mysql> INSERT INTO t VALUES (1,1);
This hangs, waiting for the lock held by the first session.
Then tail the MySQL error log, to observe the locks:
------------
TRANSACTIONS
------------
Trx id counter 3528210
Purge done for trx's n:o < 3528208 undo n:o < 0 state: running but idle
History list length 814
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 3528209, ACTIVE 30 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 360, 1 row lock(s)
MySQL thread id 1, OS thread handle 0x70000a4bd000, query id 23 localhost root update
insert into t values (1,1,null,null)
------- TRX HAS BEEN WAITING 30 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 3342 page no 3 n bits 72 index `PRIMARY` of table `test`.`t` trx id 3528209 lock mode S locks rec but not gap waiting
------------------
TABLE LOCK table `test`.`t` trx id 3528209 lock mode IX
RECORD LOCKS space id 3342 page no 3 n bits 72 index `PRIMARY` of table `test`.`t` trx id 3528209 lock mode S locks rec but not gap waiting
---TRANSACTION 3528208, ACTIVE 49 sec
2 lock struct(s), heap size 360, 1 row lock(s), undo log entries 1
MySQL thread id 2, OS thread handle 0x70000a501000, query id 17 localhost root
TABLE LOCK table `test`.`t` trx id 3528208 lock mode IX
RECORD LOCKS space id 3342 page no 3 n bits 72 index `PRIMARY` of table `test`.`t` trx id 3528208 lock_mode X locks rec but not gap

Why is there a deadlock with these two queries?

Isolation level is READ COMMITTED
The one query is looking for records where service_id is 0 ... the other is looking where service_id is NOT IN (0, ... others ... );
I'd think they would be locking different rows?
------------------------
LATEST DETECTED DEADLOCK
------------------------
2017-08-18 09:01:24 7f2d05641700
*** (1) TRANSACTION:
TRANSACTION 201694975, ACTIVE 1 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 46 lock struct(s), heap size 6544, 194 row lock(s)
MySQL thread id 33600289, OS thread handle 0x7f2d0812b700, query id 3703173090 inf-rtpctllb02-prd.rtp.netapp.com 10.60.56.150 ctl Copying to tmp table
SELECT
re.*,
r.config_id,
r.reserve_all_or_nothing,
r.owner,
r.charges
FROM
`job_charge` AS re,
`job` AS r WHERE
re.job_id = r.id AND ((re.status ='dispatched') or (re.status= 'running') or (re.status= 'held') or (re.status= 'reserved')) AND ((re.service_id ='0')) AND r.disable = 0 ORDER BY r.priority,r.id LIMIT 10000 FOR UPDATE
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1485 page no 987 n bits 104 index `PRIMARY` of table `ctl`.`job_charge` trx table locks 2 total table locks 2 trx id 201694975 lock_mode X locks rec but not gap waiting lock hold time 1 wait time before grant 0
*** (2) TRANSACTION:
TRANSACTION 201691925, ACTIVE 185 sec fetching rows
mysql tables in use 4, locked 2
1164 lock struct(s), heap size 128552, 2 row lock(s)
MySQL thread id 33599597, OS thread handle 0x7f2d05641700, query id 3703158120 inf-rtpctllb02-prd.rtp.netapp.com 10.60.56.150 ctl updating
UPDATE
`job_charge`
SET
service_id = '0'
WHERE
service_id NOT IN ('0','ctl5-staging_command-launcher.674d8c96-7c76-11e7-bc6c-ee0cf095fd00','inf-mesos-slave001.ctl.gdl.englab.netapp.com:mesos-6b256982-4ef1-4a84-ba60-58245ee7406d-S63.3987fd54-ee31-4c81-add4-4be53a6ed363:80','ctl5-staging_scheduler.912d008f-7c76-11e7-bc6c-ee0cf095fd00','ctl5-production_capacity-manager.6a869ee7-7919-11e7-bc6c-ee0cf095fd00','ctl5-production_scheduler.91de7d76-7919-11e7-bc6c-ee0cf095fd00','mysql','inf-mesos-slave001.ctl.gdl.englab.netapp.com:mesos-6b256982-4ef1-4a84-ba60-58245ee7406d-S63.48fe0555-83e9-4811-bcbc-f301da498fa6:80','ctl5-production_cleaner.6a86c5fa-7919-11e7-bc6c-ee0cf095fd00','ctl5-production_command-launcher.9f97a534-8413-11e7-bc6c-ee0cf095fd00','ctl5-production_reservation-manager.7ac1771d-7a9e-11e7-bc6c-ee0cf095fd00','ctl5-s
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1485 page no 987 n bits 104 index `PRIMARY` of table `ctl`.`job_charge` trx table locks 1 total table locks 2 trx id 201691925 lock_mode X locks rec but not gap lock hold time 13 wait time before grant 12
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1485 page no 1606 n bits 88 index `PRIMARY` of table `ctl`.`job_charge` trx table locks 1 total table locks 2 trx id 201691925 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0
*** WE ROLL BACK TRANSACTION (1)
Yes. Both must be different rows. You can see the page no 987 and 1606 as indicated in the statement
Here Transaction 1 is
SELECT
re.*,
r.config_id,
r.reserve_all_or_nothing,
r.owner,
r.charges
FROM
`job_charge` AS re,
`job` AS r WHERE
re.job_id = r.id AND ((re.status ='dispatched') or (re.status= 'running') or (re.status= 'held') or (re.status= 'reserved')) AND ((re.service_id ='0')) AND r.disable = 0 ORDER BY r.priority,r.id LIMIT 10000 FOR UPDATE
Transaction 2 is
UPDATE
`job_charge`
SET
service_id = '0'
WHERE
service_id NOT IN ('0','ctl5-staging_command-launcher.674d8c96-7c76-11e7-bc6c-ee0cf095fd00','inf-mesos-slave001.ctl.gdl.englab.netapp.com:mesos-6b256982-4ef1-4a84-ba60-58245ee7406d-S63.3987fd54-ee31-4c81-add4-4be53a6ed363:80','ctl5-staging_scheduler.912d008f-7c76-11e7-bc6c-ee0cf095fd00','ctl5-production_capacity-manager.6a869ee7-7919-11e7-bc6c-ee0cf095fd00','ctl5-production_scheduler.91de7d76-7919-11e7-bc6c-ee0cf095fd00','mysql','inf-mesos-slave001.ctl.gdl.englab.netapp.com:mesos-6b256982-4ef1-4a84-ba60-58245ee7406d-S63.48fe0555-83e9-4811-bcbc-f301da498fa6:80','ctl5-production_cleaner.6a86c5fa-7919-11e7-bc6c-ee0cf095fd00','ctl5-production_command-launcher.9f97a534-8413-11e7-bc6c-ee0cf095fd00','ctl5-production_reservation-manager.7ac1771d-7a9e-11e7-bc6c-ee0cf095fd00','ctl5-s
From the given message, we can see that transaction 1 is waiting for an Exclusive Lock (Denoted by X lock, lock needed for WRITING some value into the table) on the table 'ctl' primary key.
But at the same time, transaction 2 came into the picture which is already holding an X lock on the 'ctl' table primary key (page no 987). So, as transaction 2 is already having an X lock on 'ctl', transaction 1 can't get X lock and hence it is waiting.
But transaction 2 itself is waiting for another X lock on 'ctl' (different row from above, page no 1606). I think this row is being held by transaction 1.
So,
transaction 1 is holding a lock on a row in page no 1606 for which transaction 2 is waiting
and
transaction 2 is holding a lock on row in page no 987 for which transaction 1 is waiting
So, both are waiting for each other and hence deadlock occurred.

Ruby on rails - Transaction deadlocks with Spree

I have been getting deadlocks while creating orders in a transaction.
Spree Adjustments table acquires a lock and keeps on updating Promotions or TaxOns. In the meanwhile, other request or order needs to update the same records.
------------------------
LATEST DETECTED DEADLOCK
------------------------
2016-09-17 20:40:15 7fee358d0b00
*** (1) TRANSACTION:
TRANSACTION 3880159183, ACTIVE 2 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 10 lock struct(s), heap size 2936, 7 row lock(s), undo log entries 18
MySQL thread id 128414, OS thread handle 0x7fee6fd6ab00, query id 53800114 172.31.34.254 connect_api_user updating
UPDATE `my_variants` SET `my_variants`.`quantity` = 2, `my_variants`.`updated_at` = '2016-09-17 20:40:43' WHERE `my_variants`.`id` = 108430
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2414 page no 1809 n bits 88 index `PRIMARY` of table `marketplace`.`my_variants` trx table locks 5 total table locks 2 trx id 3880159183 lock_mode X locks rec but not gap waiting lock hold time 2 wait time before grant 0
*** (2) TRANSACTION:
TRANSACTION 3880159065, ACTIVE 3 sec starting index read
mysql tables in use 1, locked 1
22 lock struct(s), heap size 2936, 15 row lock(s), undo log entries 36
MySQL thread id 125879, OS thread handle 0x7fee358d0b00, query id 53804455 172.31.34.254 connect_api_user updating
UPDATE `spree_adjustments` SET `spree_adjustments`.`eligible` = 0 WHERE `spree_adjustments`.`adjustable_id` = 2298885 AND `spree_adjustments`.`adjustable_type` = 'Spree::LineItem' AND `spree_adjustments`.`source_type` = 'Spree::PromotionAction' AND (`spree_adjustments`.`id` != 665173)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 2414 page no 1809 n bits 88 index `PRIMARY` of table `marketplace`.`teni_variants` trx table locks 13 total table locks 2 trx id 3880159065 lock_mode X locks rec but not gap lock hold time 2 wait time before grant 0
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1544 page no 2024 n bits 432 index `index_spree_adjustments_on_adjustable_id_and_adjustable_type` of table `marketplace`.`spree_adjustments` trx table locks 13 total table locks 2 trx id 3880159065 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0
*** WE ROLL BACK TRANSACTION (1)

Baffling InnoDB deadlock

This InnoDB deadlock is really making me pull my hair. As far as I can see:
the transaction (1) is waiting for the PRIMARY on "applications"
The latter has been acquired by (2) for some rather long running updates (SELECT * FROM applications WHERE ID = xxxx FOR UPDATE)
So far, so good - one would expect (1) to wait for the lock and then get on with its work.
However, once (2) gets ready to save it's work (and commit the transaction), it fails with a deadlock since for some reason (1) has managed to get a lock on some secondary index. How the hell did (1) manage to get any locks on the row if the PRIMARY is being held by (2).
One would expect that if (2) originally acquired the PRIMARY lock (SELECT * FROM applications WHERE ID = xxxx FOR UPDATE) it would also have set locks on all the secondary indexes. Is it possible that it will not lock the "tasked" index if tasked==NULL thus allowing (1) acquire a lock on "tasked" before even getting a lock on PRIMARY?
I have had no luck replication this scenario..
Thank you!
Lauri
------------------------
LATEST DETECTED DEADLOCK
------------------------
130428 17:04:06
*** (1) TRANSACTION:
TRANSACTION A369A8C, ACTIVE 1 sec fetching rows
mysql tables in use 3, locked 3
LOCK WAIT 217 lock struct(s), heap size 31160, 636 row lock(s)
MySQL thread id 13310554, OS thread handle 0x7f06cc2d7700, query id 177699568 217.146.78.151 shard67 Sending data
SELECT `applications`.* FROM `applications`
LEFT JOIN `applicants` ON applicants.ID = applications.applicant_ID
LEFT JOIN `regions` ON regions.ID = applicants.region_ID WHERE (status <> 'Blank') AND (status <> 'Closed') AND (revised < 1367154245) AND (tasked IS NULL OR tasked < 1367147045) AND (commence_year >= '2013') AND (regions.instance_ID = '1') ORDER BY `tasked` ASC, `ID` ASC LIMIT 20 FOR UPDATE
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 20021 page no 1192 n bits 80 index `PRIMARY` of table `dream-shard67`.`applications` trx id A369A8C lock_mode X locks rec but not gap waiting
*** (2) TRANSACTION:
TRANSACTION A369A87, ACTIVE 1 sec updating or deleting
mysql tables in use 1, locked 1
16 lock struct(s), heap size 3112, 22 row lock(s), undo log entries 5
MySQL thread id 13310563, OS thread handle 0x7f06cc151700, query id 177699599 217.146.76.127 shard67
UPDATE `applications` SET `revised` = '1367157846', `tasked` = '1367157846', `revision_ID` = '140649', `xml` = 'Zms6\noMmI$%[v....snipped binary data
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 20021 page no 1192 n bits 72 index `PRIMARY` of table `dream-shard67`.`applications` trx id A369A87 lock_mode X locks rec but not gap
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 20021 page no 292 n bits 1280 index `tasked` of table `dream-shard67`.`applications` trx id A369A87 lock_mode X locks rec but not gap waiting
*** WE ROLL BACK TRANSACTION (2)
Make sure your doing most of these:
http://dev.mysql.com/doc/refman/5.0/en/innodb-deadlocks.html

reproduce such deadlock in mysql

I got information from SHOW ENGINE INNODB STATUS
*** (1) TRANSACTION:
TRANSACTION 0 2799914, ACTIVE 1 sec, process no 4106, OS thread id 139808903796480 inserting
mysql tables in use 1, locked 1
LOCK WAIT 10 lock struct(s), heap size 1216, 7 row lock(s), undo log entries 3
MySQL thread id 4284, query id 2889649 localhost 127.0.0.1 test update
INSERT INTO shipping .....
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 436366 n bits 88 index `PRIMARY` of table `testdatabase`.`order` trx id 0 2799914 lock mode S locks rec but not gap waiting
Record lock, heap no 14 PHYSICAL RECORD: n_fields 213; compact format; info bits 0
..........;
*** (2) TRANSACTION:
TRANSACTION 0 2799913, ACTIVE 1 sec, process no 4106, OS thread id 139808905824000 starting index read, thread declared inside InnoDB 500
mysql tables in use 1, locked 1
5 lock struct(s), heap size 1216, 5 row lock(s), undo log entries 4
MySQL thread id 4290, query id 2889711 localhost 127.0.0.1 test Updating
UPDATE order
........
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 436366 n bits 88 index `PRIMARY` of table `testdatabase`.`order` trx id 0 2799913 lock_mode X locks rec but not gap
Record lock, heap no 14 PHYSICAL RECORD: n_fields 213; compact format; info bits 0
..........
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 153737 n bits 88 index `PRIMARY` of table `testdatabase`.`order` trx id 0 2799913 lock_mode X locks rec but not gap waiting
Record lock, heap no 10 PHYSICAL RECORD: n_fields 213; compact format; info bits 0
......
*** WE ROLL BACK TRANSACTION (2)
------------
TRANSACTIONS
there is a FK in ship referring to order primary key.
I think T2 has hold x-lock, why it still need wait for a x-lock.
Can some one help me to reproduce such deadlock in mysql?
Thanks.
I don't know your queries, but seems like you're inserting row into child table, and then doing an update of row in parent table.
If that's true, you've hit this issue in MySQL: http://bugs.mysql.com/bug.php?id=48652
If a FOREIGN KEY constraint is defined on a table, any insert, update, or delete that requires the constraint condition to be checked sets shared record-level locks on the records that it looks at to check the constraint. InnoDB also sets these locks in the case where the constraint fails.
On a single record of the first table you have:
S lock from transaction 1 set,
S lock from transaction 2 set,
X lock from transaction 1 requested, blocked by S lock from transaction 2,
X lock from transaction 2 requested, blocked by S lock from transaction 1
Possible solution is to first update parent table, then insert row into child table. Suppose we need to increment some counter upon child row insertion, then queries would be:
UPDATE <parent row> SET count = count + 1;
INSERT <child row>; /* if the INSERT fails, roll back the trx */
If you want to update parent row only after inserting child row, you can use FOR UPDATE statement to set a lock on parent row:
SELECT <parent row> FOR UPDATE;
INSERT <child row>; /* if the INSERT fails, roll back the trx */
UPDATE <parent row> SET count = count + 1;