MySql InnoDB Repeatable Read Unexpected behavior of locks - mysql

As I know, innoDB use mechanism of consistent non blocking read,
so every transaction works with its own snapshot.
it is told also in official documentation
A consistent read does not set any locks on the tables it accesses, and therefore other sessions are free to modify those tables at the same time a consistent read is being performed on the table.
But I unexpectedly faced with behavior when classic 'read/update' deadlock appears:
Isolation level REPEATABLE READ (also is reproduced with READ COMMITTED)
Transaction 1 reads row (NOT lock in share mode).
Transaction 2 reads the same row ( ALSO NOT lock in share mode).
then
Transaction 1 tries to update this row.
Transaction 2 also tries to update this row.
After last step, innoDB detects deadlock (there is LATEST DETECTED DEADLOCK below):
----------------
2017-03-31 16:07:03 0x1f58
*** (1) TRANSACTION:
TRANSACTION 413412, ACTIVE 20 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 9 lock struct(s), heap size 1136, 6 row lock(s), undo log entries 3
MySQL thread id 33, OS thread handle 8148, query id 102005 localhost 127.0.0.1 root updating
/* update Order */ update `Order` set ... <fields to update>
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2151 page no 709 n bits 88 index PRIMARY of table `ooapp2`.`order` trx id 413412 lock_mode X locks rec but not gap waiting
Record lock, heap no 3 PHYSICAL RECORD: n_fields 54; compact format; info bits 0
*** (2) TRANSACTION:
TRANSACTION 413413, ACTIVE 11 sec starting index read
mysql tables in use 1, locked 1
9 lock struct(s), heap size 1136, 6 row lock(s), undo log entries 3
MySQL thread id 28, OS thread handle 8024, query id 102008 localhost 127.0.0.1 root updating
/* update Order */ update `Order` set ...<fields to update>
*** (2) **HOLDS THE LOCK(S):**
RECORD LOCKS space id 2151 page no 709 n bits 88 index PRIMARY of table `ooapp2`.`order` trx id 413413 lock mode S locks rec but not gap
Record lock, heap no 3 PHYSICAL RECORD: n_fields 54; compact format; info bits 0
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2151 page no 709 n bits 88 index PRIMARY of table `ooapp2`.`order` trx id 413413 lock_mode X locks rec but not gap waiting
Record lock, heap no 3 PHYSICAL RECORD: n_fields 54; compact format; info bits 0
*** WE ROLL BACK TRANSACTION (2)
I can't understand, what happens, why Transaction 2
HOLDS THE LOCK(S)
so if innoDB nevertheless not use Consistent Read with snapshot and sets S-locks
this does not correspond to the fact that is written in official manual.

Don't do that. If you might be updating a row, but meanwhile need the value, use SELECT ... FOR UPDATE;. Just do that, and forget about tx_isolation. Normally, this will turn a Deadlock into a delay. (See innodb_lock_wait_timeout, which defaults to an over-generous 50 seconds.)
Also, when you do get a Deadlock, re-run the entire transaction. Deadlocks will happen no matter how hard you try to avoid them.

Related

General error: 1205 Lock wait timeout exceeded; try restarting transaction

I am developing an app with Laravel + MySQL for online exam assessment. A lot of users submit their response at the same time (within the span for few mins) so there is heavy load on the database server.
I am getting several General error: 1205 Lock wait timeout exceeded; try restarting transaction during many concurrent submissions.
I enabled slow query log and saw following entries in my log
update `form_responses` set `photo_url` = 'https://example.com/ae252b371effc7cb11dbcbbb18602026.jpg', `form_responses`.`updated_at` = '2020-09-26 11:39:17' where `id` = 32407;
As per my understanding, this is just a simple row update with a primary column being queried (means indexing should work).
But what I can't wrap my head around is - why Innodb is not using row level locking and provide a lock on this particular row. Is there some specific Innodb value I should increase this in case?
InnoDB is using row level locking.
Problem is, the application or the ORM are using it the wrong way, not giving MySQL sufficient information on what is to be done, resulting in a deadlock.
The easiest - not, I hasten to say, the most efficient! - way to overcome this problem is probably to resort to advisory locking if you find out that the updates all happen in the same small set of methods.
Note that it is not at all a given that your deadlock is in that table; you should run SHOW ENGINE INNODB STATUS to see what the last detected deadlock was. For instance you'll find:
LATEST DETECTED DEADLOCK
------------------------
200915 11:31:03
*** (1) TRANSACTION:
TRANSACTION 22B8A76B9, ACTIVE 0 sec starting index read
mysql tables in use 3, locked 3
LOCK WAIT 4 lock struct(s), heap size 1248, 4 row lock(s)
MySQL thread id 11161737, OS thread handle 0x7fd06d708700, query id ... UPDATE table SET ... WHERE Dat_UsrId = '11254' && Dat_CreUte='...' && Dat_Tip = 'C'
*** (2) TRANSACTION:
TRANSACTION 22B8A76B4, ACTIVE 0 sec fetching rows
mysql tables in use 3, locked 3
2616 lock struct(s), heap size 244152, 7262 row lock(s)
MySQL thread id 11161676, OS thread handle 0x7fd06ca97700, query id ... UPDATE table SET ... WHERE ...
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 866959 page no 46 n bits 1616 index `NewIndex2` of table `schema`.`table` trx id 22B8A76B4 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 866959 page no 68831 n bits 72 index `PRIMARY` of table `schema`.`table` trx id 22B8A76B4 lock_mode X locks rec but not gap waiting
Record lock, heap no 5 PHYSICAL RECORD: n_fields 18; compact format; info bits 0
*** WE ROLL BACK TRANSACTION (1)
------------
Notice that here the logical error is in transaction TWO's query. This is a very strong hint:
2616 lock struct(s), heap size 244152, 7262 row lock(s)
...the WHERE needs to lock 7262 rows (!), which from the application logic's point of view is completely nonsensical (see also this for an explanation of "gapping").
Otherwise, you can resort to explicit transactions plus table level locking with LOCK TABLE form_responses WRITE.
The most efficient way to do it is to have all involved methods to enter an explicit transaction:
SET autocommit = 0;
UPDATE form_responses ... WHERE id = ...;
COMMIT WORK;

Why did this deadlock happen in MySQL?

I use JMeter to test my program, somehow the total response number is stopping increasing, then I find out there is a deadlock in MySQL. I don't understand what's meaning of below log. Seems that transaction(2) owned a S lock and trying to own a X lock of the same table. Does this cause the deadlock? If so, why will this happen?
ATEST DETECTED DEADLOCK
------------------------
2019-01-02 14:38:27 0x70000f30a000
*** (1) TRANSACTION:
TRANSACTION 24004, ACTIVE 0 sec inserting
mysql tables in use 2, locked 2
LOCK WAIT 5 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 2
MySQL thread id 11953, OS thread handle 123145549275136, query id 418447 localhost 127.0.0.1 username executing
INSERT INTO MRBS_SCHEDULE (ID,START_TIME,END_TIME,ROOM_ID,CREATE_BY,PRESIDE,REPEAT_ID,DESCRIPTION,NUM,TITLE,PRESIDE_EMAIL,PROJECTOR,CONFERENCE_CALL,CREATE_ID,BOOK_TIME,END_TYPE,EXPECTED_END_TIME) select null,'2019-01-03 19:53:00','2019-01-03 19:53:00',10113558,'d','d',12245755,'fdsfds',10,null,'d#sh.ff.com',0,0,10227622,'2019-01-02 14:38:27.358',0,'2019-01-03 19:53:00' from dual WHERE NOT EXISTS (SELECT * FROM MRBS_SCHEDULE ms where ms.START_TIME<'2019-01-03 19:53:00' and ms.END_TIME>'2019-01-03 19:53:00' and ms.ROOM_ID=10113558)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 55 page no 228 n bits 872 index IND_MRBS_SCHEDULE_END_TIME of table `meeting`.`mrbs_schedule` trx id 24004 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
*** (2) TRANSACTION:
TRANSACTION 24005, ACTIVE 0 sec inserting
mysql tables in use 2, locked 2
5 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 2
MySQL thread id 11940, OS thread handle 123145557155840, query id 418448 localhost 127.0.0.1 username executing
INSERT INTO MRBS_SCHEDULE (ID,START_TIME,END_TIME,ROOM_ID,CREATE_BY,PRESIDE,REPEAT_ID,DESCRIPTION,NUM,TITLE,PRESIDE_EMAIL,PROJECTOR,CONFERENCE_CALL,CREATE_ID,BOOK_TIME,END_TYPE,EXPECTED_END_TIME) select null,'2019-01-03 19:54:00','2019-01-03 19:54:00',10113685,'z','z',12245756,'fdsfds',10,null,'z#sz.ff.com',0,0,10227544,'2019-01-02 14:38:27.397',0,'2019-01-03 19:54:00' from dual WHERE NOT EXISTS (SELECT * FROM MRBS_SCHEDULE ms where ms.START_TIME<'2019-01-03 19:54:00' and ms.END_TIME>'2019-01-03 19:54:00' and ms.ROOM_ID=10113685)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 55 page no 228 n bits 872 index IND_MRBS_SCHEDULE_END_TIME of table `meeting`.`mrbs_schedule` trx id 24005 lock mode S
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 55 page no 228 n bits 872 index IND_MRBS_SCHEDULE_END_TIME of table `meeting`.`mrbs_schedule` trx id 24005 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
*** WE ROLL BACK TRANSACTION (2)
Edit: Table MRBS_SCHEDULE primary key use Auto_increment
Deadlock can occur anytime and it is normal behavior. It happens when transaction 1 tries to update Table B using Table A but at the same time Transaction 2 tries to update Table A using Table B.
basicly Transactions holds theirselves and since nobody wants to give up, deadlock occurs.
you can set deadlock threshold for your db using innodb_lock_wait_timeout
From the documentation,
When deadlock detection is enabled (the default), InnoDB automatically detects transaction deadlocks and rolls back a transaction or transactions to break the deadlock. InnoDB tries to pick small transactions to roll back, where the size of a transaction is determined by the number of rows inserted, updated, or deleted.
InnoDB is aware of table locks if innodb_table_locks = 1 (the default) and autocommit = 0, and the MySQL layer above it knows about row-level locks. Otherwise, InnoDB cannot detect deadlocks where a table lock set by a MySQL LOCK TABLES statement or a lock set by a storage engine other than InnoDB is involved. Resolve these situations by setting the value of the innodb_lock_wait_timeout system variable.
Reference: https://dev.mysql.com/doc/refman/8.0/en/innodb-deadlock-detection.html

I recently upgraded to mysql 5.7 and observing frequent deadlock issue. Is there any change related to locking in 5.7?

I recently upgraded to mysql 5.7 and observing frequent deadlock issue. Is there any change related to locking 5.7 because I didn't experienced this problem before.
I have a simple table called retry_records having auto incremented id as a primary key and a column unique_reference as an unique index and two more columns. This table is getting used in fully multi threaded environment where a section of code is selecting n records from this table, process the data and update this table in case of process failure and other section of code is deleting few processed records from the table and another section of code is inserting a new record in this table. There is no join in any query, no batch insert, batch update, batch delete except select. All the update/delete/insert transaction has single statement. Even then I am observing deadlock on insertion.
Below are the logs:
------------------------
LATEST DETECTED DEADLOCK
------------------------
2016-09-22 13:05:09 0x7f3f427e0700
*** (1) TRANSACTION:
TRANSACTION 39420432534, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1
MySQL thread id 37503, OS thread handle 139917976340224, query id 1695822465
Insert Query..
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1334 page no 1726952 n bits 312 index id of table `database_name`.`table_name` trx id 39420432534 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
*** (2) TRANSACTION:
TRANSACTION 39420432524, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
6 lock struct(s), heap size 1136, 5 row lock(s), undo log entries 1
MySQL thread id 37494, OS thread handle 139909675222784, query id 1695822438
Another insert query...
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1334 page no 1726952 n bits 312 index id of table
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1334 page no 1726952 n bits 312 index id of table
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
I experienced similar issue. Since 5.7, there is a new parameter name innodb_deadlock_detect
This option is used to disable deadlock detection. On high concurrency
systems, deadlock detection can cause a slowdown when numerous threads
wait for the same lock. At times, it may be more efficient to disable
deadlock detection and rely on the innodb_lock_wait_timeout setting
for transaction rollback when a deadlock occurs.
Disabling it could improve your performance and reduce deadlock issue.

Why does this InnoDB deadlock occur on a simple UPDATE?

We have a Symphony2 web application and use Doctrine as ORM.
Backend we use AWS RDS MySQL with the InnoDB engine.
SHOW VARIABLES LIKE "%version%";
innodb_version 5.6.22
protocol_version 10
slave_type_conversions
version 5.6.22-log
version_comment MySQL Community Server (GPL)
version_compile_machine x86_64
version_compile_os Linux
We can trigger an unexpected deadlock on the database server by clicking repeatedly on a submit button.
This is part of the error message we get:
------------------------
LATEST DETECTED DEADLOCK
------------------------
2015-06-16 07:13:17 2aebdd723700
*** (1) TRANSACTION:
TRANSACTION 2009024, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 360, 1 row lock(s), undo log entries 2
MySQL thread id 11125, OS thread handle 0x2aebdd6e2700, query id 1666530 172.31.26.135 db_name updating
UPDATE cmp_item SET title = 'Contract of employment 456', date_modification = '2015-06-16 09:14:37' WHERE id = 64297
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 15706 page no 10780 n bits 80 index `PRIMARY` of table `db_name`.`cmp_item` trx id 2009024 lock_mode X locks rec but not gap waiting
Record lock, heap no 14 PHYSICAL RECORD: n_fields 102; compact format; info bits 0
*** (2) TRANSACTION:
TRANSACTION 2009010, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
12 lock struct(s), heap size 2936, 7 row lock(s), undo log entries 4
MySQL thread id 11124, OS thread handle 0x2aebdd723700, query id 1666536 172.31.26.135 db_name updating
UPDATE cmp_item SET title = 'Contract of employment 456', date_modification = '2015-06-16 09:14:38' WHERE id = 64297
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 15706 page no 10780 n bits 80 index `PRIMARY` of table `db_name`.`cmp_item` trx id 2009010 lock mode S locks rec but not gap
Record lock, heap no 14 PHYSICAL RECORD: n_fields 102; compact format; info bits 0
We don't understand why this deadlock would occur.
Why does it happen, and how can we avoid this?
A deadlock usually (always?) requires multiple statements to have been executed in each of two (or more) threads. Presumably your two Updates were each preceded by some other statements that locked various rows in various tables. Look through the entire BEGIN...COMMIT for each. Show them to us.

reproduce such deadlock in mysql

I got information from SHOW ENGINE INNODB STATUS
*** (1) TRANSACTION:
TRANSACTION 0 2799914, ACTIVE 1 sec, process no 4106, OS thread id 139808903796480 inserting
mysql tables in use 1, locked 1
LOCK WAIT 10 lock struct(s), heap size 1216, 7 row lock(s), undo log entries 3
MySQL thread id 4284, query id 2889649 localhost 127.0.0.1 test update
INSERT INTO shipping .....
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 436366 n bits 88 index `PRIMARY` of table `testdatabase`.`order` trx id 0 2799914 lock mode S locks rec but not gap waiting
Record lock, heap no 14 PHYSICAL RECORD: n_fields 213; compact format; info bits 0
..........;
*** (2) TRANSACTION:
TRANSACTION 0 2799913, ACTIVE 1 sec, process no 4106, OS thread id 139808905824000 starting index read, thread declared inside InnoDB 500
mysql tables in use 1, locked 1
5 lock struct(s), heap size 1216, 5 row lock(s), undo log entries 4
MySQL thread id 4290, query id 2889711 localhost 127.0.0.1 test Updating
UPDATE order
........
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 436366 n bits 88 index `PRIMARY` of table `testdatabase`.`order` trx id 0 2799913 lock_mode X locks rec but not gap
Record lock, heap no 14 PHYSICAL RECORD: n_fields 213; compact format; info bits 0
..........
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 153737 n bits 88 index `PRIMARY` of table `testdatabase`.`order` trx id 0 2799913 lock_mode X locks rec but not gap waiting
Record lock, heap no 10 PHYSICAL RECORD: n_fields 213; compact format; info bits 0
......
*** WE ROLL BACK TRANSACTION (2)
------------
TRANSACTIONS
there is a FK in ship referring to order primary key.
I think T2 has hold x-lock, why it still need wait for a x-lock.
Can some one help me to reproduce such deadlock in mysql?
Thanks.
I don't know your queries, but seems like you're inserting row into child table, and then doing an update of row in parent table.
If that's true, you've hit this issue in MySQL: http://bugs.mysql.com/bug.php?id=48652
If a FOREIGN KEY constraint is defined on a table, any insert, update, or delete that requires the constraint condition to be checked sets shared record-level locks on the records that it looks at to check the constraint. InnoDB also sets these locks in the case where the constraint fails.
On a single record of the first table you have:
S lock from transaction 1 set,
S lock from transaction 2 set,
X lock from transaction 1 requested, blocked by S lock from transaction 2,
X lock from transaction 2 requested, blocked by S lock from transaction 1
Possible solution is to first update parent table, then insert row into child table. Suppose we need to increment some counter upon child row insertion, then queries would be:
UPDATE <parent row> SET count = count + 1;
INSERT <child row>; /* if the INSERT fails, roll back the trx */
If you want to update parent row only after inserting child row, you can use FOR UPDATE statement to set a lock on parent row:
SELECT <parent row> FOR UPDATE;
INSERT <child row>; /* if the INSERT fails, roll back the trx */
UPDATE <parent row> SET count = count + 1;