We have a Symphony2 web application and use Doctrine as ORM.
Backend we use AWS RDS MySQL with the InnoDB engine.
SHOW VARIABLES LIKE "%version%";
innodb_version 5.6.22
protocol_version 10
slave_type_conversions
version 5.6.22-log
version_comment MySQL Community Server (GPL)
version_compile_machine x86_64
version_compile_os Linux
We can trigger an unexpected deadlock on the database server by clicking repeatedly on a submit button.
This is part of the error message we get:
------------------------
LATEST DETECTED DEADLOCK
------------------------
2015-06-16 07:13:17 2aebdd723700
*** (1) TRANSACTION:
TRANSACTION 2009024, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 360, 1 row lock(s), undo log entries 2
MySQL thread id 11125, OS thread handle 0x2aebdd6e2700, query id 1666530 172.31.26.135 db_name updating
UPDATE cmp_item SET title = 'Contract of employment 456', date_modification = '2015-06-16 09:14:37' WHERE id = 64297
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 15706 page no 10780 n bits 80 index `PRIMARY` of table `db_name`.`cmp_item` trx id 2009024 lock_mode X locks rec but not gap waiting
Record lock, heap no 14 PHYSICAL RECORD: n_fields 102; compact format; info bits 0
*** (2) TRANSACTION:
TRANSACTION 2009010, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
12 lock struct(s), heap size 2936, 7 row lock(s), undo log entries 4
MySQL thread id 11124, OS thread handle 0x2aebdd723700, query id 1666536 172.31.26.135 db_name updating
UPDATE cmp_item SET title = 'Contract of employment 456', date_modification = '2015-06-16 09:14:38' WHERE id = 64297
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 15706 page no 10780 n bits 80 index `PRIMARY` of table `db_name`.`cmp_item` trx id 2009010 lock mode S locks rec but not gap
Record lock, heap no 14 PHYSICAL RECORD: n_fields 102; compact format; info bits 0
We don't understand why this deadlock would occur.
Why does it happen, and how can we avoid this?
A deadlock usually (always?) requires multiple statements to have been executed in each of two (or more) threads. Presumably your two Updates were each preceded by some other statements that locked various rows in various tables. Look through the entire BEGIN...COMMIT for each. Show them to us.
Related
I am developing an app with Laravel + MySQL for online exam assessment. A lot of users submit their response at the same time (within the span for few mins) so there is heavy load on the database server.
I am getting several General error: 1205 Lock wait timeout exceeded; try restarting transaction during many concurrent submissions.
I enabled slow query log and saw following entries in my log
update `form_responses` set `photo_url` = 'https://example.com/ae252b371effc7cb11dbcbbb18602026.jpg', `form_responses`.`updated_at` = '2020-09-26 11:39:17' where `id` = 32407;
As per my understanding, this is just a simple row update with a primary column being queried (means indexing should work).
But what I can't wrap my head around is - why Innodb is not using row level locking and provide a lock on this particular row. Is there some specific Innodb value I should increase this in case?
InnoDB is using row level locking.
Problem is, the application or the ORM are using it the wrong way, not giving MySQL sufficient information on what is to be done, resulting in a deadlock.
The easiest - not, I hasten to say, the most efficient! - way to overcome this problem is probably to resort to advisory locking if you find out that the updates all happen in the same small set of methods.
Note that it is not at all a given that your deadlock is in that table; you should run SHOW ENGINE INNODB STATUS to see what the last detected deadlock was. For instance you'll find:
LATEST DETECTED DEADLOCK
------------------------
200915 11:31:03
*** (1) TRANSACTION:
TRANSACTION 22B8A76B9, ACTIVE 0 sec starting index read
mysql tables in use 3, locked 3
LOCK WAIT 4 lock struct(s), heap size 1248, 4 row lock(s)
MySQL thread id 11161737, OS thread handle 0x7fd06d708700, query id ... UPDATE table SET ... WHERE Dat_UsrId = '11254' && Dat_CreUte='...' && Dat_Tip = 'C'
*** (2) TRANSACTION:
TRANSACTION 22B8A76B4, ACTIVE 0 sec fetching rows
mysql tables in use 3, locked 3
2616 lock struct(s), heap size 244152, 7262 row lock(s)
MySQL thread id 11161676, OS thread handle 0x7fd06ca97700, query id ... UPDATE table SET ... WHERE ...
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 866959 page no 46 n bits 1616 index `NewIndex2` of table `schema`.`table` trx id 22B8A76B4 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 866959 page no 68831 n bits 72 index `PRIMARY` of table `schema`.`table` trx id 22B8A76B4 lock_mode X locks rec but not gap waiting
Record lock, heap no 5 PHYSICAL RECORD: n_fields 18; compact format; info bits 0
*** WE ROLL BACK TRANSACTION (1)
------------
Notice that here the logical error is in transaction TWO's query. This is a very strong hint:
2616 lock struct(s), heap size 244152, 7262 row lock(s)
...the WHERE needs to lock 7262 rows (!), which from the application logic's point of view is completely nonsensical (see also this for an explanation of "gapping").
Otherwise, you can resort to explicit transactions plus table level locking with LOCK TABLE form_responses WRITE.
The most efficient way to do it is to have all involved methods to enter an explicit transaction:
SET autocommit = 0;
UPDATE form_responses ... WHERE id = ...;
COMMIT WORK;
I use JMeter to test my program, somehow the total response number is stopping increasing, then I find out there is a deadlock in MySQL. I don't understand what's meaning of below log. Seems that transaction(2) owned a S lock and trying to own a X lock of the same table. Does this cause the deadlock? If so, why will this happen?
ATEST DETECTED DEADLOCK
------------------------
2019-01-02 14:38:27 0x70000f30a000
*** (1) TRANSACTION:
TRANSACTION 24004, ACTIVE 0 sec inserting
mysql tables in use 2, locked 2
LOCK WAIT 5 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 2
MySQL thread id 11953, OS thread handle 123145549275136, query id 418447 localhost 127.0.0.1 username executing
INSERT INTO MRBS_SCHEDULE (ID,START_TIME,END_TIME,ROOM_ID,CREATE_BY,PRESIDE,REPEAT_ID,DESCRIPTION,NUM,TITLE,PRESIDE_EMAIL,PROJECTOR,CONFERENCE_CALL,CREATE_ID,BOOK_TIME,END_TYPE,EXPECTED_END_TIME) select null,'2019-01-03 19:53:00','2019-01-03 19:53:00',10113558,'d','d',12245755,'fdsfds',10,null,'d#sh.ff.com',0,0,10227622,'2019-01-02 14:38:27.358',0,'2019-01-03 19:53:00' from dual WHERE NOT EXISTS (SELECT * FROM MRBS_SCHEDULE ms where ms.START_TIME<'2019-01-03 19:53:00' and ms.END_TIME>'2019-01-03 19:53:00' and ms.ROOM_ID=10113558)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 55 page no 228 n bits 872 index IND_MRBS_SCHEDULE_END_TIME of table `meeting`.`mrbs_schedule` trx id 24004 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
*** (2) TRANSACTION:
TRANSACTION 24005, ACTIVE 0 sec inserting
mysql tables in use 2, locked 2
5 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 2
MySQL thread id 11940, OS thread handle 123145557155840, query id 418448 localhost 127.0.0.1 username executing
INSERT INTO MRBS_SCHEDULE (ID,START_TIME,END_TIME,ROOM_ID,CREATE_BY,PRESIDE,REPEAT_ID,DESCRIPTION,NUM,TITLE,PRESIDE_EMAIL,PROJECTOR,CONFERENCE_CALL,CREATE_ID,BOOK_TIME,END_TYPE,EXPECTED_END_TIME) select null,'2019-01-03 19:54:00','2019-01-03 19:54:00',10113685,'z','z',12245756,'fdsfds',10,null,'z#sz.ff.com',0,0,10227544,'2019-01-02 14:38:27.397',0,'2019-01-03 19:54:00' from dual WHERE NOT EXISTS (SELECT * FROM MRBS_SCHEDULE ms where ms.START_TIME<'2019-01-03 19:54:00' and ms.END_TIME>'2019-01-03 19:54:00' and ms.ROOM_ID=10113685)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 55 page no 228 n bits 872 index IND_MRBS_SCHEDULE_END_TIME of table `meeting`.`mrbs_schedule` trx id 24005 lock mode S
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 55 page no 228 n bits 872 index IND_MRBS_SCHEDULE_END_TIME of table `meeting`.`mrbs_schedule` trx id 24005 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
*** WE ROLL BACK TRANSACTION (2)
Edit: Table MRBS_SCHEDULE primary key use Auto_increment
Deadlock can occur anytime and it is normal behavior. It happens when transaction 1 tries to update Table B using Table A but at the same time Transaction 2 tries to update Table A using Table B.
basicly Transactions holds theirselves and since nobody wants to give up, deadlock occurs.
you can set deadlock threshold for your db using innodb_lock_wait_timeout
From the documentation,
When deadlock detection is enabled (the default), InnoDB automatically detects transaction deadlocks and rolls back a transaction or transactions to break the deadlock. InnoDB tries to pick small transactions to roll back, where the size of a transaction is determined by the number of rows inserted, updated, or deleted.
InnoDB is aware of table locks if innodb_table_locks = 1 (the default) and autocommit = 0, and the MySQL layer above it knows about row-level locks. Otherwise, InnoDB cannot detect deadlocks where a table lock set by a MySQL LOCK TABLES statement or a lock set by a storage engine other than InnoDB is involved. Resolve these situations by setting the value of the innodb_lock_wait_timeout system variable.
Reference: https://dev.mysql.com/doc/refman/8.0/en/innodb-deadlock-detection.html
As I know, innoDB use mechanism of consistent non blocking read,
so every transaction works with its own snapshot.
it is told also in official documentation
A consistent read does not set any locks on the tables it accesses, and therefore other sessions are free to modify those tables at the same time a consistent read is being performed on the table.
But I unexpectedly faced with behavior when classic 'read/update' deadlock appears:
Isolation level REPEATABLE READ (also is reproduced with READ COMMITTED)
Transaction 1 reads row (NOT lock in share mode).
Transaction 2 reads the same row ( ALSO NOT lock in share mode).
then
Transaction 1 tries to update this row.
Transaction 2 also tries to update this row.
After last step, innoDB detects deadlock (there is LATEST DETECTED DEADLOCK below):
----------------
2017-03-31 16:07:03 0x1f58
*** (1) TRANSACTION:
TRANSACTION 413412, ACTIVE 20 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 9 lock struct(s), heap size 1136, 6 row lock(s), undo log entries 3
MySQL thread id 33, OS thread handle 8148, query id 102005 localhost 127.0.0.1 root updating
/* update Order */ update `Order` set ... <fields to update>
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2151 page no 709 n bits 88 index PRIMARY of table `ooapp2`.`order` trx id 413412 lock_mode X locks rec but not gap waiting
Record lock, heap no 3 PHYSICAL RECORD: n_fields 54; compact format; info bits 0
*** (2) TRANSACTION:
TRANSACTION 413413, ACTIVE 11 sec starting index read
mysql tables in use 1, locked 1
9 lock struct(s), heap size 1136, 6 row lock(s), undo log entries 3
MySQL thread id 28, OS thread handle 8024, query id 102008 localhost 127.0.0.1 root updating
/* update Order */ update `Order` set ...<fields to update>
*** (2) **HOLDS THE LOCK(S):**
RECORD LOCKS space id 2151 page no 709 n bits 88 index PRIMARY of table `ooapp2`.`order` trx id 413413 lock mode S locks rec but not gap
Record lock, heap no 3 PHYSICAL RECORD: n_fields 54; compact format; info bits 0
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2151 page no 709 n bits 88 index PRIMARY of table `ooapp2`.`order` trx id 413413 lock_mode X locks rec but not gap waiting
Record lock, heap no 3 PHYSICAL RECORD: n_fields 54; compact format; info bits 0
*** WE ROLL BACK TRANSACTION (2)
I can't understand, what happens, why Transaction 2
HOLDS THE LOCK(S)
so if innoDB nevertheless not use Consistent Read with snapshot and sets S-locks
this does not correspond to the fact that is written in official manual.
Don't do that. If you might be updating a row, but meanwhile need the value, use SELECT ... FOR UPDATE;. Just do that, and forget about tx_isolation. Normally, this will turn a Deadlock into a delay. (See innodb_lock_wait_timeout, which defaults to an over-generous 50 seconds.)
Also, when you do get a Deadlock, re-run the entire transaction. Deadlocks will happen no matter how hard you try to avoid them.
I recently upgraded to mysql 5.7 and observing frequent deadlock issue. Is there any change related to locking 5.7 because I didn't experienced this problem before.
I have a simple table called retry_records having auto incremented id as a primary key and a column unique_reference as an unique index and two more columns. This table is getting used in fully multi threaded environment where a section of code is selecting n records from this table, process the data and update this table in case of process failure and other section of code is deleting few processed records from the table and another section of code is inserting a new record in this table. There is no join in any query, no batch insert, batch update, batch delete except select. All the update/delete/insert transaction has single statement. Even then I am observing deadlock on insertion.
Below are the logs:
------------------------
LATEST DETECTED DEADLOCK
------------------------
2016-09-22 13:05:09 0x7f3f427e0700
*** (1) TRANSACTION:
TRANSACTION 39420432534, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1
MySQL thread id 37503, OS thread handle 139917976340224, query id 1695822465
Insert Query..
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1334 page no 1726952 n bits 312 index id of table `database_name`.`table_name` trx id 39420432534 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
*** (2) TRANSACTION:
TRANSACTION 39420432524, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
6 lock struct(s), heap size 1136, 5 row lock(s), undo log entries 1
MySQL thread id 37494, OS thread handle 139909675222784, query id 1695822438
Another insert query...
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1334 page no 1726952 n bits 312 index id of table
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1334 page no 1726952 n bits 312 index id of table
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
I experienced similar issue. Since 5.7, there is a new parameter name innodb_deadlock_detect
This option is used to disable deadlock detection. On high concurrency
systems, deadlock detection can cause a slowdown when numerous threads
wait for the same lock. At times, it may be more efficient to disable
deadlock detection and rely on the innodb_lock_wait_timeout setting
for transaction rollback when a deadlock occurs.
Disabling it could improve your performance and reduce deadlock issue.
Using 'show engine innodb status' I see that wordpress has two deadlocks. I'd like to clear these up but I don't see an active process for either of these cmds (IE something to 'kill' and hopefully force a rollback).
I can see thread ids, query ids, etc but nothing that I can use to stop either job.
Suggestions on how to resolve this?
EDIT: Here's the (relevant?) portion of the status:
------------------------
LATEST DETECTED DEADLOCK
------------------------
110327 10:54:14
*** (1) TRANSACTION:
TRANSACTION 9FBA099E, ACTIVE 0 sec, process no 14207, OS thread id 1228433728 starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 12505112, query id 909492800 juno....edu 129....54 wordpress_user updating
DELETE FROM wp_options WHERE option_name = ''_site_transient_timeout_theme_roots''
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 4951009 page no 4 n bits 384 index `option_name` of table `wordpress_work`.`wp_options` trx id 9FBA099E lock_mode X waiting
Record lock, heap no 309 PHYSICAL RECORD: n_fields 2; compact format; info bits 32
0: len 30; hex 5f736974655f7472616e7369656e745f74696d656f75745f7468656d655f; asc _site_transient_timeout_theme_; (total 35 bytes);
1: len 8; hex 0000000000002b6d; asc +m;;
*** (2) TRANSACTION:
TRANSACTION 9FBA0995, ACTIVE 0 sec, process no 14207, OS thread id 1230031168 starting index read
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1248, 2 row lock(s)
MySQL thread id 12505095, query id 909492789 juno....edu 129.....54 wordpress_user updating
DELETE FROM wp_options WHERE option_name = ''_site_transient_timeout_theme_roots''
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 4951009 page no 4 n bits 384 index `option_name` of table `wordpress_work`.`wp_options` trx id 9FBA0995 lock_mode X locks rec but not gap
Record lock, heap no 309 PHYSICAL RECORD: n_fields 2; compact format; info bits 32
0: len 30; hex 5f736974655f7472616e7369656e745f74696d656f75745f7468656d655f; asc _site_transient_timeout_theme_; (total 35 bytes);
1: len 8; hex 0000000000002b6d; asc +m;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 4951009 page no 4 n bits 384 index `option_name` of table `wordpress_work`.`wp_options` trx id 9FBA0995 lock_mode X waiting
Record lock, heap no 309 PHYSICAL RECORD: n_fields 2; compact format; info bits 32
0: len 30; hex 5f736974655f7472616e7369656e745f74696d656f75745f7468656d655f; asc _site_transient_timeout_theme_; (total 35 bytes);
1: len 8; hex 0000000000002b6d; asc +m;;
*** WE ROLL BACK TRANSACTION (1)
Given some 'innodb status' output like this:
---TRANSACTION 0 0, not started, process no 1024, OS thread id 140386055603968
MySQL thread id 197, query id 771 localhost marc
show innodb status
you'd want to do
KILL QUERY 771
to kill one of the two queries that are deadlocked. That'll kill the query, but leave the connection open. if you want to kill the connection, then you'd do KILL 197.
Using 'show engine innodb status' I see that wordpress has two deadlocks... Suggestions on how to resolve this?
We were seeing Java hibernate issues causing stuck locks. We found the locks by combing trough the output from:
show engine innodb status;
This spits out a crap-ton of information. The relevant section is in the TRANSACTIONS section. In your output the relevant problem seems to be:
3 lock struct(s), heap size 1248, 2 row lock(s)
MySQL thread id 12505095, query id 909492789 juno....edu 129.....54
For us it was the # lock struct(s) that indicated a stuck lock. To kill it you need to execute by using the "thread id #" specified -- in this case:
kill 12505095
This worked on AWS MySQL RDS as well as local MySQL.
In our TRANSACTIONS section we also see the following:
---TRANSACTION 644793773, ACTIVE 21 sec
2 lock struct(s), heap size 360, 1 row lock(s)
MySQL thread id 217, OS thread handle 0x2aef097700, query id 1177 1.3.5.7 mpsp cleaning up
We look for both the 2 lock struct(s) and the ACTIVE 21 sec messages.
I know this is old, but typically when you see something like this it's because a deadlock occurred and the app that triggered the deadlock has long since moved on -- the victim of the deadlock got warned and either failed, or logged an error or retried, and either way has moved on to other productive things. You usually don't need to do anything other than look into the cause of the deadlock and try and avoid future deadlocks, if you're writing the software. If you're just using the software (e.g. Wordpress if you don't work at Wordpress), you could report the deadlock as a possible bug.