mysql innodb_locks table dump to file - mysql

According to mysql reference about InnoDB Transaction and Locking Information page:
The data exposed by the transaction and locking tables (INNODB_TRX, INNODB_LOCKS, and INNODB_LOCK_WAITS) represents a glimpse into fast-changing data.
For performance reasons, and to minimize the chance of misleading joins between the transaction and locking tables, InnoDB collects the required transaction and locking information into an intermediate buffer whenever a SELECT on any of the tables is issued.
So I'd like to know if there is a way to write this info/buffer into a file?

I don't think there's any way to access the buffer you speak of directly. You can use SELECT queries against the named INFORMATION_SCHEMA tables, which will read from those buffers indirectly.
An alternative is to make the output of SHOW ENGINE INNODB STATUS be dumped to the MySQL Server error log every 15 seconds. You can optionally make this status include lock information. See https://dev.mysql.com/doc/refman/5.6/en/innodb-enabling-monitors.html
Example: I enabled the InnoDB lock monitor with lock options.
mysql> set global innodb_status_output_locks=on;
mysql> set global innodb_status_output=on;
Then I created one transaction, inserting into a test table. But don't commit yet.
mysql> BEGIN;
mysql> INSERT INTO t VALUES (1,1);
In a second window, I begin another transaction, with another insert designed to conflict with the first one.
mysql> INSERT INTO t VALUES (1,1);
This hangs, waiting for the lock held by the first session.
Then tail the MySQL error log, to observe the locks:
------------
TRANSACTIONS
------------
Trx id counter 3528210
Purge done for trx's n:o < 3528208 undo n:o < 0 state: running but idle
History list length 814
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 3528209, ACTIVE 30 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 360, 1 row lock(s)
MySQL thread id 1, OS thread handle 0x70000a4bd000, query id 23 localhost root update
insert into t values (1,1,null,null)
------- TRX HAS BEEN WAITING 30 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 3342 page no 3 n bits 72 index `PRIMARY` of table `test`.`t` trx id 3528209 lock mode S locks rec but not gap waiting
------------------
TABLE LOCK table `test`.`t` trx id 3528209 lock mode IX
RECORD LOCKS space id 3342 page no 3 n bits 72 index `PRIMARY` of table `test`.`t` trx id 3528209 lock mode S locks rec but not gap waiting
---TRANSACTION 3528208, ACTIVE 49 sec
2 lock struct(s), heap size 360, 1 row lock(s), undo log entries 1
MySQL thread id 2, OS thread handle 0x70000a501000, query id 17 localhost root
TABLE LOCK table `test`.`t` trx id 3528208 lock mode IX
RECORD LOCKS space id 3342 page no 3 n bits 72 index `PRIMARY` of table `test`.`t` trx id 3528208 lock_mode X locks rec but not gap

Related

MySQL (InnoDB) deadlock

We are struggling with one deadlock which is occuring multiple times a day in our production environment.
------------------------
LATEST DETECTED DEADLOCK
------------------------
2018-12-27 19:07:34 7fcef1959700
*** (1) TRANSACTION:
TRANSACTION 2125001468, ACTIVE 2 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 1
MySQL thread id 42190185, OS thread handle 0x7fcffc0b1700, query id 918842488 --- updating
UPDATE synchronization SET service_synchronized_at = NULL WHERE id = 116212
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 615 page no 288 n bits 528 index `PRIMARY` of table `app`.`synchronization` trx table locks 1 total table locks 2 trx id 2125001468 lock_mode X locks rec but not gap waiting lock hold time 2 wait time before grant 0
*** (2) TRANSACTION:
TRANSACTION 2125001355, ACTIVE 5 sec fetching rows
mysql tables in use 2, locked 2
25216 lock struct(s), heap size 3683880, 5297668 row lock(s), undo log entries 94
MySQL thread id 42189517, OS thread handle 0x7fcef1959700, query id 918842042 --- updating
UPDATE synchronization s SET s.service_synchronized_at = now() WHERE s.service_synchronized_at IS NULL AND s.user_id IN (* time consuming select to determine which users should be updated *)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 615 page no 288 n bits 528 index `PRIMARY` of table `app`.`synchronization` trx table locks 2 total table locks 2 trx id 2125001355 lock_mode X lock hold time 3 wait time before grant 0
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 615 page no 2313 n bits 472 index `PRIMARY` of table `app`.`synchronization` trx table locks 2 total table locks 2 trx id 2125001355 lock_mode X waiting lock hold time 0 wait time before grant 0
*** WE ROLL BACK TRANSACTION (1)
In one query I want to set service_synchronized_at = NULL. service_synchronized_at is always not null in first query before update.
In second query I add a where condition s.service_synchronized_at IS NULL thinking it would result in non-locking rows with not null values. Guess I was wrong.
The table has only primary index on id and unique constraint on user_id (and of course foreign key on user_id).
Any help is welcome.

Why is there a deadlock with these two queries?

Isolation level is READ COMMITTED
The one query is looking for records where service_id is 0 ... the other is looking where service_id is NOT IN (0, ... others ... );
I'd think they would be locking different rows?
------------------------
LATEST DETECTED DEADLOCK
------------------------
2017-08-18 09:01:24 7f2d05641700
*** (1) TRANSACTION:
TRANSACTION 201694975, ACTIVE 1 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 46 lock struct(s), heap size 6544, 194 row lock(s)
MySQL thread id 33600289, OS thread handle 0x7f2d0812b700, query id 3703173090 inf-rtpctllb02-prd.rtp.netapp.com 10.60.56.150 ctl Copying to tmp table
SELECT
re.*,
r.config_id,
r.reserve_all_or_nothing,
r.owner,
r.charges
FROM
`job_charge` AS re,
`job` AS r WHERE
re.job_id = r.id AND ((re.status ='dispatched') or (re.status= 'running') or (re.status= 'held') or (re.status= 'reserved')) AND ((re.service_id ='0')) AND r.disable = 0 ORDER BY r.priority,r.id LIMIT 10000 FOR UPDATE
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1485 page no 987 n bits 104 index `PRIMARY` of table `ctl`.`job_charge` trx table locks 2 total table locks 2 trx id 201694975 lock_mode X locks rec but not gap waiting lock hold time 1 wait time before grant 0
*** (2) TRANSACTION:
TRANSACTION 201691925, ACTIVE 185 sec fetching rows
mysql tables in use 4, locked 2
1164 lock struct(s), heap size 128552, 2 row lock(s)
MySQL thread id 33599597, OS thread handle 0x7f2d05641700, query id 3703158120 inf-rtpctllb02-prd.rtp.netapp.com 10.60.56.150 ctl updating
UPDATE
`job_charge`
SET
service_id = '0'
WHERE
service_id NOT IN ('0','ctl5-staging_command-launcher.674d8c96-7c76-11e7-bc6c-ee0cf095fd00','inf-mesos-slave001.ctl.gdl.englab.netapp.com:mesos-6b256982-4ef1-4a84-ba60-58245ee7406d-S63.3987fd54-ee31-4c81-add4-4be53a6ed363:80','ctl5-staging_scheduler.912d008f-7c76-11e7-bc6c-ee0cf095fd00','ctl5-production_capacity-manager.6a869ee7-7919-11e7-bc6c-ee0cf095fd00','ctl5-production_scheduler.91de7d76-7919-11e7-bc6c-ee0cf095fd00','mysql','inf-mesos-slave001.ctl.gdl.englab.netapp.com:mesos-6b256982-4ef1-4a84-ba60-58245ee7406d-S63.48fe0555-83e9-4811-bcbc-f301da498fa6:80','ctl5-production_cleaner.6a86c5fa-7919-11e7-bc6c-ee0cf095fd00','ctl5-production_command-launcher.9f97a534-8413-11e7-bc6c-ee0cf095fd00','ctl5-production_reservation-manager.7ac1771d-7a9e-11e7-bc6c-ee0cf095fd00','ctl5-s
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1485 page no 987 n bits 104 index `PRIMARY` of table `ctl`.`job_charge` trx table locks 1 total table locks 2 trx id 201691925 lock_mode X locks rec but not gap lock hold time 13 wait time before grant 12
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1485 page no 1606 n bits 88 index `PRIMARY` of table `ctl`.`job_charge` trx table locks 1 total table locks 2 trx id 201691925 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0
*** WE ROLL BACK TRANSACTION (1)
Yes. Both must be different rows. You can see the page no 987 and 1606 as indicated in the statement
Here Transaction 1 is
SELECT
re.*,
r.config_id,
r.reserve_all_or_nothing,
r.owner,
r.charges
FROM
`job_charge` AS re,
`job` AS r WHERE
re.job_id = r.id AND ((re.status ='dispatched') or (re.status= 'running') or (re.status= 'held') or (re.status= 'reserved')) AND ((re.service_id ='0')) AND r.disable = 0 ORDER BY r.priority,r.id LIMIT 10000 FOR UPDATE
Transaction 2 is
UPDATE
`job_charge`
SET
service_id = '0'
WHERE
service_id NOT IN ('0','ctl5-staging_command-launcher.674d8c96-7c76-11e7-bc6c-ee0cf095fd00','inf-mesos-slave001.ctl.gdl.englab.netapp.com:mesos-6b256982-4ef1-4a84-ba60-58245ee7406d-S63.3987fd54-ee31-4c81-add4-4be53a6ed363:80','ctl5-staging_scheduler.912d008f-7c76-11e7-bc6c-ee0cf095fd00','ctl5-production_capacity-manager.6a869ee7-7919-11e7-bc6c-ee0cf095fd00','ctl5-production_scheduler.91de7d76-7919-11e7-bc6c-ee0cf095fd00','mysql','inf-mesos-slave001.ctl.gdl.englab.netapp.com:mesos-6b256982-4ef1-4a84-ba60-58245ee7406d-S63.48fe0555-83e9-4811-bcbc-f301da498fa6:80','ctl5-production_cleaner.6a86c5fa-7919-11e7-bc6c-ee0cf095fd00','ctl5-production_command-launcher.9f97a534-8413-11e7-bc6c-ee0cf095fd00','ctl5-production_reservation-manager.7ac1771d-7a9e-11e7-bc6c-ee0cf095fd00','ctl5-s
From the given message, we can see that transaction 1 is waiting for an Exclusive Lock (Denoted by X lock, lock needed for WRITING some value into the table) on the table 'ctl' primary key.
But at the same time, transaction 2 came into the picture which is already holding an X lock on the 'ctl' table primary key (page no 987). So, as transaction 2 is already having an X lock on 'ctl', transaction 1 can't get X lock and hence it is waiting.
But transaction 2 itself is waiting for another X lock on 'ctl' (different row from above, page no 1606). I think this row is being held by transaction 1.
So,
transaction 1 is holding a lock on a row in page no 1606 for which transaction 2 is waiting
and
transaction 2 is holding a lock on row in page no 987 for which transaction 1 is waiting
So, both are waiting for each other and hence deadlock occurred.

MySql InnoDB Repeatable Read Unexpected behavior of locks

As I know, innoDB use mechanism of consistent non blocking read,
so every transaction works with its own snapshot.
it is told also in official documentation
A consistent read does not set any locks on the tables it accesses, and therefore other sessions are free to modify those tables at the same time a consistent read is being performed on the table.
But I unexpectedly faced with behavior when classic 'read/update' deadlock appears:
Isolation level REPEATABLE READ (also is reproduced with READ COMMITTED)
Transaction 1 reads row (NOT lock in share mode).
Transaction 2 reads the same row ( ALSO NOT lock in share mode).
then
Transaction 1 tries to update this row.
Transaction 2 also tries to update this row.
After last step, innoDB detects deadlock (there is LATEST DETECTED DEADLOCK below):
----------------
2017-03-31 16:07:03 0x1f58
*** (1) TRANSACTION:
TRANSACTION 413412, ACTIVE 20 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 9 lock struct(s), heap size 1136, 6 row lock(s), undo log entries 3
MySQL thread id 33, OS thread handle 8148, query id 102005 localhost 127.0.0.1 root updating
/* update Order */ update `Order` set ... <fields to update>
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2151 page no 709 n bits 88 index PRIMARY of table `ooapp2`.`order` trx id 413412 lock_mode X locks rec but not gap waiting
Record lock, heap no 3 PHYSICAL RECORD: n_fields 54; compact format; info bits 0
*** (2) TRANSACTION:
TRANSACTION 413413, ACTIVE 11 sec starting index read
mysql tables in use 1, locked 1
9 lock struct(s), heap size 1136, 6 row lock(s), undo log entries 3
MySQL thread id 28, OS thread handle 8024, query id 102008 localhost 127.0.0.1 root updating
/* update Order */ update `Order` set ...<fields to update>
*** (2) **HOLDS THE LOCK(S):**
RECORD LOCKS space id 2151 page no 709 n bits 88 index PRIMARY of table `ooapp2`.`order` trx id 413413 lock mode S locks rec but not gap
Record lock, heap no 3 PHYSICAL RECORD: n_fields 54; compact format; info bits 0
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2151 page no 709 n bits 88 index PRIMARY of table `ooapp2`.`order` trx id 413413 lock_mode X locks rec but not gap waiting
Record lock, heap no 3 PHYSICAL RECORD: n_fields 54; compact format; info bits 0
*** WE ROLL BACK TRANSACTION (2)
I can't understand, what happens, why Transaction 2
HOLDS THE LOCK(S)
so if innoDB nevertheless not use Consistent Read with snapshot and sets S-locks
this does not correspond to the fact that is written in official manual.
Don't do that. If you might be updating a row, but meanwhile need the value, use SELECT ... FOR UPDATE;. Just do that, and forget about tx_isolation. Normally, this will turn a Deadlock into a delay. (See innodb_lock_wait_timeout, which defaults to an over-generous 50 seconds.)
Also, when you do get a Deadlock, re-run the entire transaction. Deadlocks will happen no matter how hard you try to avoid them.

Baffling InnoDB deadlock

This InnoDB deadlock is really making me pull my hair. As far as I can see:
the transaction (1) is waiting for the PRIMARY on "applications"
The latter has been acquired by (2) for some rather long running updates (SELECT * FROM applications WHERE ID = xxxx FOR UPDATE)
So far, so good - one would expect (1) to wait for the lock and then get on with its work.
However, once (2) gets ready to save it's work (and commit the transaction), it fails with a deadlock since for some reason (1) has managed to get a lock on some secondary index. How the hell did (1) manage to get any locks on the row if the PRIMARY is being held by (2).
One would expect that if (2) originally acquired the PRIMARY lock (SELECT * FROM applications WHERE ID = xxxx FOR UPDATE) it would also have set locks on all the secondary indexes. Is it possible that it will not lock the "tasked" index if tasked==NULL thus allowing (1) acquire a lock on "tasked" before even getting a lock on PRIMARY?
I have had no luck replication this scenario..
Thank you!
Lauri
------------------------
LATEST DETECTED DEADLOCK
------------------------
130428 17:04:06
*** (1) TRANSACTION:
TRANSACTION A369A8C, ACTIVE 1 sec fetching rows
mysql tables in use 3, locked 3
LOCK WAIT 217 lock struct(s), heap size 31160, 636 row lock(s)
MySQL thread id 13310554, OS thread handle 0x7f06cc2d7700, query id 177699568 217.146.78.151 shard67 Sending data
SELECT `applications`.* FROM `applications`
LEFT JOIN `applicants` ON applicants.ID = applications.applicant_ID
LEFT JOIN `regions` ON regions.ID = applicants.region_ID WHERE (status <> 'Blank') AND (status <> 'Closed') AND (revised < 1367154245) AND (tasked IS NULL OR tasked < 1367147045) AND (commence_year >= '2013') AND (regions.instance_ID = '1') ORDER BY `tasked` ASC, `ID` ASC LIMIT 20 FOR UPDATE
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 20021 page no 1192 n bits 80 index `PRIMARY` of table `dream-shard67`.`applications` trx id A369A8C lock_mode X locks rec but not gap waiting
*** (2) TRANSACTION:
TRANSACTION A369A87, ACTIVE 1 sec updating or deleting
mysql tables in use 1, locked 1
16 lock struct(s), heap size 3112, 22 row lock(s), undo log entries 5
MySQL thread id 13310563, OS thread handle 0x7f06cc151700, query id 177699599 217.146.76.127 shard67
UPDATE `applications` SET `revised` = '1367157846', `tasked` = '1367157846', `revision_ID` = '140649', `xml` = 'Zms6\noMmI$%[v....snipped binary data
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 20021 page no 1192 n bits 72 index `PRIMARY` of table `dream-shard67`.`applications` trx id A369A87 lock_mode X locks rec but not gap
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 20021 page no 292 n bits 1280 index `tasked` of table `dream-shard67`.`applications` trx id A369A87 lock_mode X locks rec but not gap waiting
*** WE ROLL BACK TRANSACTION (2)
Make sure your doing most of these:
http://dev.mysql.com/doc/refman/5.0/en/innodb-deadlocks.html

reproduce such deadlock in mysql

I got information from SHOW ENGINE INNODB STATUS
*** (1) TRANSACTION:
TRANSACTION 0 2799914, ACTIVE 1 sec, process no 4106, OS thread id 139808903796480 inserting
mysql tables in use 1, locked 1
LOCK WAIT 10 lock struct(s), heap size 1216, 7 row lock(s), undo log entries 3
MySQL thread id 4284, query id 2889649 localhost 127.0.0.1 test update
INSERT INTO shipping .....
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 436366 n bits 88 index `PRIMARY` of table `testdatabase`.`order` trx id 0 2799914 lock mode S locks rec but not gap waiting
Record lock, heap no 14 PHYSICAL RECORD: n_fields 213; compact format; info bits 0
..........;
*** (2) TRANSACTION:
TRANSACTION 0 2799913, ACTIVE 1 sec, process no 4106, OS thread id 139808905824000 starting index read, thread declared inside InnoDB 500
mysql tables in use 1, locked 1
5 lock struct(s), heap size 1216, 5 row lock(s), undo log entries 4
MySQL thread id 4290, query id 2889711 localhost 127.0.0.1 test Updating
UPDATE order
........
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 436366 n bits 88 index `PRIMARY` of table `testdatabase`.`order` trx id 0 2799913 lock_mode X locks rec but not gap
Record lock, heap no 14 PHYSICAL RECORD: n_fields 213; compact format; info bits 0
..........
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 153737 n bits 88 index `PRIMARY` of table `testdatabase`.`order` trx id 0 2799913 lock_mode X locks rec but not gap waiting
Record lock, heap no 10 PHYSICAL RECORD: n_fields 213; compact format; info bits 0
......
*** WE ROLL BACK TRANSACTION (2)
------------
TRANSACTIONS
there is a FK in ship referring to order primary key.
I think T2 has hold x-lock, why it still need wait for a x-lock.
Can some one help me to reproduce such deadlock in mysql?
Thanks.
I don't know your queries, but seems like you're inserting row into child table, and then doing an update of row in parent table.
If that's true, you've hit this issue in MySQL: http://bugs.mysql.com/bug.php?id=48652
If a FOREIGN KEY constraint is defined on a table, any insert, update, or delete that requires the constraint condition to be checked sets shared record-level locks on the records that it looks at to check the constraint. InnoDB also sets these locks in the case where the constraint fails.
On a single record of the first table you have:
S lock from transaction 1 set,
S lock from transaction 2 set,
X lock from transaction 1 requested, blocked by S lock from transaction 2,
X lock from transaction 2 requested, blocked by S lock from transaction 1
Possible solution is to first update parent table, then insert row into child table. Suppose we need to increment some counter upon child row insertion, then queries would be:
UPDATE <parent row> SET count = count + 1;
INSERT <child row>; /* if the INSERT fails, roll back the trx */
If you want to update parent row only after inserting child row, you can use FOR UPDATE statement to set a lock on parent row:
SELECT <parent row> FOR UPDATE;
INSERT <child row>; /* if the INSERT fails, roll back the trx */
UPDATE <parent row> SET count = count + 1;