is there any way to show MySQL table lock history - mysql

I am using MySQL 5.7.26,is there any way to show table lock history?I know how to check current table lock using:
show OPEN TABLES where In_use > 0;
I want to avoid table lock in my SQL query,so view the history may help.Current lock may not get all locked table(Maybe the locked table query not execute right now).

Today I found a way to show this in MySQL 5.7:
show engine innodb status
this is the part of result output:
====================================
2020-09-25 10:36:53 0x7ff4b4c5d700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 12 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 762213 srv_active, 0 srv_shutdown, 139997 srv_idle
srv_master_thread log flush and writes: 902210
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 363879865
OS WAIT ARRAY INFO: signal count 448819672
RW-shared spins 0, rounds 425739262, OS waits 209673117
RW-excl spins 0, rounds 4271714439, OS waits 133598673
RW-sx spins 135926, rounds 2757000, OS waits 52788
Spin rounds per wait: 425739262.00 RW-shared, 4271714439.00 RW-excl, 20.28 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2020-09-25 10:10:41 0x7ff4b4ce1700
*** (1) TRANSACTION:
TRANSACTION 841893940, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s)
MySQL thread id 108348, OS thread handle 140688980530944, query id 184994716 172.19.104.233 root Searching rows for update
UPDATE r_room_seat SET status = 1,
id = (SELECT #update_id := id),
user_id = 62659,
robot_flag = 1
WHERE room_play_id = 539 and status = 0 and online = 1
LIMIT 1
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 3493 page no 528 n bits 176 index PRIMARY of table `room_db`.`r_room_seat` trx id 841893940 lock_mode X locks rec but not gap waiting
Record lock, heap no 56 PHYSICAL RECORD: n_fields 20; compact format; info bits 0
0: len 8; hex 000000000000d649; asc I;;
1: len 6; hex 0000322d9b9a; asc 2- ;;
2: len 7; hex 3e000040042eda; asc > # . ;;
3: len 8; hex 800000000000021b; asc ;;
4: len 8; hex 8000000000000097; asc ;;
5: len 8; hex 800000000000f4d5; asc ;;
6: len 8; hex 7fffffffffffffff; asc ;;
7: len 8; hex 8000017452aa2c5e; asc tR ,^;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 80000000; asc ;;
10: len 8; hex 8000017452aa2c5e; asc tR ,^;;
11: len 8; hex 8000017452aa2c5e; asc tR ,^;;
12: len 4; hex 80000001; asc ;;
13: len 1; hex 81; asc ;;
14: len 8; hex 8000000000000005; asc ;;
15: len 10; hex 48415050595f5241494e; asc HAPPY_RAIN;;
16: len 4; hex 80000001; asc ;;
17: len 8; hex 8000000000000000; asc ;;
18: len 8; hex 8000000000000074; asc t;;
19: len 4; hex 80000001; asc ;;
*** (2) TRANSACTION:
TRANSACTION 841893905, ACTIVE 0 sec fetching rows
mysql tables in use 2, locked 2
746 lock struct(s), heap size 90320, 128960 row lock(s)
MySQL thread id 100540, OS thread handle 140688982152960, query id 184994612 172.19.104.233 root Sending data
UPDATE r_room_seat s
SET s.status = 1,
s.room_play_id = (SELECT #update_id := max(room_play_id)),
s.user_id = (
case when s.seat_num = 1
then 63182
when s.seat_num = 2
then 16472
else -1 end
),
s.robot_flag = (
case when s.seat_num = 1
then 1
when s.seat_num = 2
then 0
else 0 end
)
WHERE s.status = 0
and s.online = 1
and s.room_play_id in (
select room_play_id
from (
select room_play_id as room_play_id
from r_room_seat
where room_id = 166
and status = 0
group by room_play_id
having count(*) = 2
limit 1
) a
)
and s.room_id = 166
LIMIT 2
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 3493 page no 528 n bits 176 index PRIMARY of table `room_db`.`r_room_seat` trx id 841893905 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
you can get the dead lock information and analysis.

Related

Unexpected database locking in symfony

I have a recurring database lock that I'm trying to understand.
I have a cron that is adding jobs to a queue. These jobs are then processed by various workers on different instances of production. All of them are importing data from a resource, and then adding/updating a table. Theoretically, there would never be two jobs writing to the same row, as the data is partitioned in such a way that this won't happen (jobs are created per provider, which means the id used in this table is unique).
This is what msyql is logging for the lock:
2023-01-12T10:49:41.876818Z 610974 [Note] InnoDB:
*** (1) TRANSACTION:
TRANSACTION 41621347924, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1
MySQL thread id 610972, OS thread handle 22890357896960, query id 17518666 {instance} {app} updating
UPDATE offer SET timestamp = '2023-01-12 10:49:41' WHERE id = 17479128
2023-01-12T10:49:41.876850Z 610974 [Note] InnoDB: *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 3037 page no 11514 n bits 96 index PRIMARY of table `{app}_production`.`offer` trx id 41621347924 lock_mode X locks rec but not gap waiting
Record lock, heap no 16 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 810ab5d8; asc ;;
1: len 6; hex 0009b0d35e55; asc ^U;;
2: len 7; hex 230002401d04c1; asc # # ;;
3: len 4; hex 8000e54d; asc M;;
4: len 17; hex 3330313032393431313035333030323630; asc 30102941105300260;;
5: len 3; hex 8fce3c; asc <;;
6: len 3; hex 8fce41; asc A;;
7: len 30; hex 3c7265736f75726365207265736f7572636569643d223330313032393431; asc <resource resourceid="30102941; (total 326 bytes);
8: len 1; hex 81; asc ;;
9: len 5; hex 99af18ac69; asc i;;
2023-01-12T10:49:41.877237Z 610974 [Note] InnoDB: *** (2) TRANSACTION:
TRANSACTION 41621347925, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1
MySQL thread id 610974, OS thread handle 22894496118528, query id 17518667 {instance} {app} updating
UPDATE offer SET timestamp = '2023-01-12 10:49:41' WHERE id = 17821328
2023-01-12T10:49:41.877262Z 610974 [Note] InnoDB: *** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 3037 page no 11514 n bits 96 index PRIMARY of table `{app}_production`.`offer` trx id 41621347925 lock_mode X locks rec but not gap
Record lock, heap no 16 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 810ab5d8; asc ;;
1: len 6; hex 0009b0d35e55; asc ^U;;
2: len 7; hex 230002401d04c1; asc # # ;;
3: len 4; hex 8000e54d; asc M;;
4: len 17; hex 3330313032393431313035333030323630; asc 30102941105300260;;
5: len 3; hex 8fce3c; asc <;;
6: len 3; hex 8fce41; asc A;;
7: len 30; hex 3c7265736f75726365207265736f7572636569643d223330313032393431; asc <resource resourceid="30102941; (total 326 bytes);
8: len 1; hex 81; asc ;;
9: len 5; hex 99af18ac69; asc i;;
2023-01-12T10:49:41.877643Z 610974 [Note] InnoDB: *** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 3037 page no 10300 n bits 104 index PRIMARY of table `{app}_production`.`offer` trx id 41621347925 lock_mode X locks rec but not gap waiting
Record lock, heap no 4 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 810fee90; asc ;;
1: len 6; hex 0009b0d35e54; asc ^T;;
2: len 7; hex 22000240070d5b; asc " # [;;
3: len 4; hex 8000e54d; asc M;;
4: len 17; hex 3330313032393431313035333030323630; asc 30102941105300260;;
5: len 3; hex 8fce38; asc 8;;
6: len 3; hex 8fce3c; asc <;;
7: len 30; hex 3c7265736f75726365207265736f7572636569643d223330313032393431; asc <resource resourceid="30102941; (total 326 bytes);
8: len 1; hex 81; asc ;;
9: len 5; hex 99af18ac69; asc i;;
2023-01-12T10:49:41.878014Z 610974 [Note] InnoDB: *** WE ROLL BACK TRANSACTION (2)
I have always sucked at sorting out deadlocking issues so looking for a bit of guidance.
Is this a table or a row lock? I'm using symfony 2.8 (don't ask) and it's doctrine packages.
Is there some way to disable the locking completely?
What can I read to better understand these logs?

In Read uncommitted transaction, select statement hold S lock

I changed my select statement transaction to read uncommitted but still it holds shared lock and get involved in deadlock with another update statement,
Select statements are in whatsnew(uid,friend_news_id,actor_news_id) SP
SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED ;
set #actor_news_limit = 2;
set #player_news_limit = 6;
if (select
n.id
from
news n
inner join
news_user_xrf nu
on
n.id = nu.news_id
inner join
friends f
on
nu.user_id = f.friend_id
inner join
user u
on
u.id = f.friend_id
where
f.user_id = uid
and
n.id > friend_news_id
order by
n.points desc,n.newsDate desc
limit 0,1) is null
then
select * from
(select
n.id,
'whatshot_friendsnews' as type,
n.content,
'0' as actor_id,
u.id as player_id,
n.newsType,
n.like_text,
n.link,
n.newsDate,
u.imgurl,
u.firstname,
u.lastname,
n.points
from
news n
inner join
news_user_xrf nu
on
n.id = nu.news_id
inner join
user u
on
u.id = nu.user_id
where
u.id <> uid
and
n.id > friend_news_id
order by
n.newsDate desc
limit 0,3) a order by newsDate;
else
select * from
(select
n.id,
'whatshot_friendsnews' as type,
n.content,
'0' as actor_id,
u.id as player_id,
n.newsType,
n.like_text,
n.link,
n.newsDate,
u.imgurl,
u.firstname,
u.lastname,
n.points
from
news n
inner join
news_user_xrf nu
on
n.id = nu.news_id
inner join
friends f
on
nu.user_id = f.friend_id
inner join
user u
on
u.id = f.friend_id
where
f.user_id = uid
and
n.id > friend_news_id
order by
n.points desc,n.newsDate desc
limit 0,3) a order by newsDate;
end if;
select * from
(select
n.id,
'whatshot_actornews' as type,
n.content,
na.actor_id,
"0" as player_id,
n.newsType,
n.like_text,
n.link,
n.newsDate,
n.newsImg as imgurl,
'' as firstname,
'' as lastname,
n.points
from
news n
inner join
news_actor_xrf na
on
n.id = na.news_id
where
newsType = 2
and
n.id > actor_news_id
order by
points desc,
newsDate desc
limit
0,3
) a order by newsDate;
COMMIT ;
END
now i have another update statement where i update same table and this statement wait for X lock but it cannot have it as the whatsnew sp holds S lock on the same table,
below is Update sp
set SQL_SAFE_UPDATES=0;
update user
set networth = (select sum(p.avgcostprice * p.quantity)
from portfolio p where p.user_id = user.id) +
(select truncate(sum((a.currentvalue - p.avgcostprice)*p.quantity),2)
from actor a inner join portfolio p on a.id = p.actor_id where p.user_id = user.id) +
user.balance;
update user u
set u.rank = (select count(*) from friends f
where ifnull(friend_networth,0) > user_networth
and f.user_id = u.id) + 1;
update user u
set u.rank_text = (select concat(u.rank,'/',(select
count(*)+1 from friends f where f.user_id = u.id)));
Now both of these stored procedure gets involved in deadlock, but question is even when i am using read uncommitted why select statement
holds Shared lock on the rows, it should read( dirty read) from the
table. here is SHOW ENGINE INNODB STATUS output
=====================================
2014-07-06 03:44:57 2ad677681700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 2 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 32811 srv_active, 0 srv_shutdown, 89228 srv_idle
srv_master_thread log flush and writes: 122039
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 22814
OS WAIT ARRAY INFO: signal count 21778
Mutex spin waits 6627, rounds 508985, OS waits 15885
RW-shared spins 5876, rounds 176305, OS waits 5659
RW-excl spins 44, rounds 38079, OS waits 1237
Spin rounds per wait: 76.80 mutex, 30.00 RW-shared, 865.43 RW-excl
------------------------
LATEST DETECTED DEADLOCK
------------------------
2014-07-06 02:01:41 2ad677681700
*** (1) TRANSACTION:
TRANSACTION 1925770, ACTIVE 0 sec fetching rows
mysql tables in use 2, locked 2
LOCK WAIT 25 lock struct(s), heap size 3112, 909 row lock(s)
MySQL thread id 28709, OS thread handle 0x2ad677640700, query id 349542 ec2-54-226-145-7.compute-1.amazonaws.com 54.227.116.3 Database1 Searching rows for update
update user u
set u.rank_text = (select concat(u.rank,''/'',(select
count(*)+1 from friends f where f.user_id = u.id)))
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 19303 page no 29 n bits 104 index `PRIMARY` of table `Database1`.`user` trx id 1925770 lock_mode X waiting
Record lock, heap no 31 PHYSICAL RECORD: n_fields 25; compact format; info bits 0
0: len 8; hex 8000000000000416; asc ;;
1: len 6; hex 0000001d214a; asc !J;;
2: len 7; hex 6f0000018d08f9; asc o ;;
3: len 8; hex 80005af4001478e8; asc Z x ;;
4: len 5; hex 5269736869; asc Rishi;;
5: len 5; hex 506174656c; asc Patel;;
6: len 23; hex 7269736869706174656c5f3939407961686f6f2e636f6d; asc rishi_9999#yahoo.com;;
7: len 30; hex 43414146336348376866544942414a434844466b5a42664937544a554643; asc CAAF3cH7hfTIBAJCHDFkZBfI7TJUFC; (total 201 bytes);
8: len 8; hex 800000005405f359; asc T Y;;
9: len 9; hex 80000000000007d619; asc ;;
10: len 1; hex 01; asc ;;
11: len 30; hex 313034383539303739377c3130303030303937303735313938317c313030; asc 1048590797|100000970751981|100; (total 58 bytes);
12: len 30; hex 68747470733a2f2f67726170682e66616365626f6f6b2e636f6d2f313030; asc https://graph.facebook.com/100; (total 50 bytes);
13: SQL NULL;
14: len 4; hex 80000002; asc ;;
15: len 4; hex 466f4145; asc FoAE;;
16: SQL NULL;
17: len 4; hex 00000000; asc ;;
18: len 3; hex 322f35; asc 2/5;;
19: SQL NULL;
20: len 8; hex 8000000000000007; asc ;;
21: len 4; hex 05c58045; asc E;;
22: SQL NULL;
23: len 8; hex 8000000000000000; asc ;;
24: len 8; hex 8000000000000007; asc ;;
*** (2) TRANSACTION:
TRANSACTION 1925761, ACTIVE 22 sec starting index read
mysql tables in use 4, locked 4
14 lock struct(s), heap size 3112, 4 row lock(s)
MySQL thread id 28723, OS thread handle 0x2ad677681700, query id 349517 ec2-54-91-34-208.compute-1.amazonaws.com 54.91.34.208 Database1 Sending data
call whatsnew(uid,friend_news_id,actor_news_id)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 19303 page no 29 n bits 104 index `PRIMARY` of table `Database1`.`user` trx id 1925761 lock mode S locks rec but not gap
Record lock, heap no 31 PHYSICAL RECORD: n_fields 25; compact format; info bits 0
0: len 8; hex 8000000000000416; asc ;;
1: len 6; hex 0000001d214a; asc !J;;
2: len 7; hex 6f0000018d08f9; asc o ;;
3: len 8; hex 80005af4001478e8; asc Z x ;;
4: len 5; hex 5269736869; asc Rishi;;
5: len 5; hex 506174656c; asc Patel;;
6: len 23; hex 7269736869706174656c5f3939407961686f6f2e636f6d; asc rishi_9999#yahoo.com;;
7: len 30; hex 43414146336348376866544942414a434844466b5a42664937544a554643; asc CAAF3cH7hfTIBAJCHDFkZBfI7TJUFC; (total 201 bytes);
8: len 8; hex 800000005405f359; asc T Y;;
9: len 9; hex 80000000000007d619; asc ;;
10: len 1; hex 01; asc ;;
11: len 30; hex 313034383539303739377c3130303030303937303735313938317c313030; asc 1048590797|100000970751981|100; (total 58 bytes);
12: len 30; hex 68747470733a2f2f67726170682e66616365626f6f6b2e636f6d2f313030; asc https://graph.facebook.com/100; (total 50 bytes);
13: SQL NULL;
14: len 4; hex 80000002; asc ;;
15: len 4; hex 466f4145; asc FoAE;;
16: SQL NULL;
17: len 4; hex 00000000; asc ;;
18: len 3; hex 322f35; asc 2/5;;
19: SQL NULL;
20: len 8; hex 8000000000000007; asc ;;
21: len 4; hex 05c58045; asc E;;
22: SQL NULL;
23: len 8; hex 8000000000000000; asc ;;
24: len 8; hex 8000000000000007; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 19303 page no 29 n bits 104 index `PRIMARY` of table `Database1`.`user` trx id 1925761 lock mode S locks rec but not gap waiting
Record lock, heap no 26 PHYSICAL RECORD: n_fields 25; compact format; info bits 0
0: len 8; hex 800000000000040b; asc ;;
1: len 6; hex 0000001d214a; asc !J;;
2: len 7; hex 6f0000018d0878; asc o x;;
3: len 8; hex 80005af401790e35; asc Z y 5;;
4: len 3; hex 52616a; asc Raj;;
5: len 5; hex 506174656c; asc Patel;;
6: len 22; hex 706174656c3039303972616a407961686f6f2e636f6d; asc patel#yahoo.com;;
7: len 30; hex 434141463363483768665449424145726535315545495475473766556652; asc CAAF3cH7hfTIBAEre51UEITuG7fUfR; (total 199 bytes);
8: len 8; hex 8000000054060653; asc T S;;
9: len 9; hex 800000000000098440; asc #;;
10: len 1; hex 01; asc ;;
11: len 30; hex 313034383539303739377c313432343530303835307c3130303030303937; asc 1048590797|1424500850|10000097; (total 85 bytes);
12: len 30; hex 68747470733a2f2f67726170682e66616365626f6f6b2e636f6d2f313030; asc graph.facebook.com/10000; (total 50 bytes);
13: SQL NULL;
14: len 4; hex 80000001; asc ;;
15: SQL NULL;
16: SQL NULL;
17: len 4; hex 00000000; asc ;;
18: len 3; hex 312f37; asc 1/7;;
19: len 4; hex 00000000; asc ;;
20: len 8; hex 8000000000000018; asc ;;
21: len 4; hex 6bdb7e45; asc k ~E;;
22: SQL NULL;
23: len 8; hex 8000000000000000; asc ;;
24: len 8; hex 8000000000000018; asc ;;
*** WE ROLL BACK TRANSACTION (2)
------------
TRANSACTIONS
------------
Trx id counter 1929046
Purge done for trx''s n:o < 1929045 undo n:o < 0 state: running but idle
History list length 1711
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 28864, OS thread handle 0x2ad677681700, query id 357530 122.169.237.65 Database1 init
SHOW ENGINE INNODB STATUS
---TRANSACTION 1929043, not started
MySQL thread id 28863, OS thread handle 0x2ad677640700, query id 357529 ec2-54-226-145-7.compute-1.amazonaws.com 54.227.116.3 Database1 creating table
create table temp_ceiling (actor_id int, p_change float, cval float, fval float)
---TRANSACTION 1929042, not started
MySQL thread id 28859, OS thread handle 0x2ad675ee3700, query id 357510 localhost 127.0.0.1 rdsadmin cleaning up
---TRANSACTION 1929045, ACTIVE 0 sec committing foreign key definitions
5 lock struct(s), heap size 1248, 0 row lock(s), undo log entries 8
MySQL thread id 28863, OS thread handle 0x2ad677640700, query id 357529 ec2-54-226-145-7.compute-1.amazonaws.com 54.227.116.3 Database1 creating table
create table temp_ceiling (actor_id int, p_change float, cval float, fval float)
TABLE LOCK table `SYS_TABLES` trx id 1929045 lock mode IX
TABLE LOCK table `SYS_COLUMNS` trx id 1929045 lock mode IX
TABLE LOCK table `SYS_TABLESPACES` trx id 1929045 lock mode IX
TABLE LOCK table `SYS_DATAFILES` trx id 1929045 lock mode IX
TABLE LOCK table `SYS_INDEXES` trx id 1929045 lock mode IX
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o''s: 0, sync i/o''s: 0
Pending flushes (fsync) log: 0; buffer pool: 0
946 OS file reads, 122080 OS file writes, 61052 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 4000.00 writes/s, 5000.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 612301, node heap has 33 buffer(s)
0.00 hash searches/s, 5.00 non-hash searches/s
---
LOG
---
Log sequence number 502059361
Log flushed up to 502055320
Pages flushed up to 502053286
Last checkpoint at 502053286
0 pending log writes, 0 pending chkp writes
39359 log i/o''s done, 1.00 log i/o''s/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 316588032; in additional pool allocated 0
Dictionary memory allocated 269802
Buffer pool size 18880
Free buffers 11148
Database pages 7699
Old database pages 2822
Modified db pages 16
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 886, created 6813, written 73085
0.00 reads/s, 4000.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 7699, unzip_LRU len: 0
I/O sum[0]:cur[1], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Main thread process no. 18631, id 47100585178880, state: sleeping
Number of rows inserted 43271, updated 151830, deleted 2009, read 7297650700
0.00 inserts/s, 14.49 updates/s, 0.00 deletes/s, 167000.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
'
please help

MySQL deadlocking issue with InnoDB

I have a central database server and several "worker" servers which are executing queries like this concurrently:
UPDATE job_queue
SET
worker = '108.166.81.112',
attempts = attempts + 1,
started = '2014-01-14 10:34:03',
token = '13eb3e6a8c3e1becb34051e08f19fd62'
WHERE completed = '0000-00-00 00:00:00'
AND (started = '0000-00-00 00:00:00' OR started < '2014-01-14 10:29:03')
AND attempts < 2
ORDER BY priority DESC, inserted
LIMIT 1
Occasionally my job_queue table locks up and if I run "SHOW ENGINE INNODB STATUS", I get something like this:
------------------------
LATEST DETECTED DEADLOCK
------------------------
140114 10:34:15
*** (1) TRANSACTION:
TRANSACTION 0 46984514, ACTIVE 0 sec, process no 590, OS thread id 140366633146112 fetching rows
mysql tables in use 1, locked 1
LOCK WAIT 20 lock struct(s), heap size 3024, 545 row lock(s)
MySQL thread id 677401, query id 19385205 10.179.103.110 root init
UPDATE job_queue SET worker='108.166.81.112', attempts=attempts+1, started='2014-01-14 10:34:03', token='13eb3e6a8c3e1becb34051e08f19fd62' WHERE completed='0000-00-00 00:00:00' AND (started='0000-00-00 00:00:00' OR started<'2014-01-14 10:29:03') AND attempts<2 ORDER BY priority DESC, inserted LIMIT 1
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 245767 n bits 128 index `PRIMARY` of table `database`.`job_queue` trx id 0 46984514 lock_mode X waiting
Record lock, heap no 34 PHYSICAL RECORD: n_fields 12; compact format; info bits 0
0: len 3; hex 800210; asc ;; 1: len 6; hex 000002cced25; asc %;; 2: len 7; hex 000003c00f1970; asc p;; 3: len 30; hex 4f3a31343a2243425343616368654170704a6f62223a363a7b733a31393a; asc O:14:"CBSCacheAppJob":6:{s:19:;...(truncated); 4: len 1; hex 80; asc ;; 5: len 8; hex 800012513c58bf24; asc Q<X $;; 6: len 8; hex 800012513c58cc17; asc Q<X ;; 7: len 14; hex 31302e3137392e3130332e313333; asc 10.179.103.133;; 8: len 1; hex 81; asc ;; 9: len 8; hex 800012513c58cc32; asc Q<X 2;; 10: len 0; hex ; asc ;; 11: len 30; hex 353264393033616162656634346239626536306463346438666432303066; asc 52d903aabef44b9be60dc4d8fd200f;...(truncated);
*** (2) TRANSACTION:
TRANSACTION 0 46984485, ACTIVE 17 sec, process no 590, OS thread id 140366633547520 starting index read, thread declared inside InnoDB 500
mysql tables in use 1, locked 1
4 lock struct(s), heap size 1216, 2 row lock(s), undo log entries 1
MySQL thread id 676723, query id 19385209 10.179.103.133 root init
UPDATE job_queue SET worker='10.179.103.133', attempts=attempts+1, started='2014-01-14 10:34:03', token='efd21d0d34f44badbc30386db4dd252e' WHERE completed='0000-00-00 00:00:00' AND (started='0000-00-00 00:00:00' OR started<'2014-01-14 10:29:03') AND attempts<2 ORDER BY priority DESC, inserted LIMIT 1
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 245767 n bits 128 index `PRIMARY` of table `database`.`job_queue` trx id 0 46984485 lock_mode X locks rec but not gap
Record lock, heap no 34 PHYSICAL RECORD: n_fields 12; compact format; info bits 0
0: len 3; hex 800210; asc ;; 1: len 6; hex 000002cced25; asc %;; 2: len 7; hex 000003c00f1970; asc p;; 3: len 30; hex 4f3a31343a2243425343616368654170704a6f62223a363a7b733a31393a; asc O:14:"CBSCacheAppJob":6:{s:19:;...(truncated); 4: len 1; hex 80; asc ;; 5: len 8; hex 800012513c58bf24; asc Q<X $;; 6: len 8; hex 800012513c58cc17; asc Q<X ;; 7: len 14; hex 31302e3137392e3130332e313333; asc 10.179.103.133;; 8: len 1; hex 81; asc ;; 9: len 8; hex 800012513c58cc32; asc Q<X 2;; 10: len 0; hex ; asc ;; 11: len 30; hex 353264393033616162656634346239626536306463346438666432303066; asc 52d903aabef44b9be60dc4d8fd200f;...(truncated);
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 57 n bits 120 index `PRIMARY` of table `database`.`job_queue` trx id 0 46984485 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 12; compact format; info bits 0
0: len 3; hex 800001; asc ;; 1: len 6; hex 000002ccdab1; asc ;; 2: len 7; hex 000003c0352b3f; asc 5+?;; 3: len 30; hex 4f3a31323a224175746f50696c6f744a6f62223a363a7b733a31383a2200; asc O:12:"AutoPilotJob":6:{s:18:" ;...(truncated); 4: len 1; hex 82; asc ;; 5: len 8; hex 800012513c58af57; asc Q<X W;; 6: len 8; hex 800012513c58bf22; asc Q<X ";; 7: len 14; hex 3130382e3136362e38312e313132; asc 108.166.81.112;; 8: len 1; hex 81; asc ;; 9: len 8; hex 800012513c58bf23; asc Q<X #;; 10: len 0; hex ; asc ;; 11: len 30; hex 616331376430346339326163613366323330646164323239363764336266; asc ac17d04c92aca3f230dad22967d3bf;...(truncated);
*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------
Trx id counter 0 46989905
Purge done for trx's n:o < 0 46986227 undo n:o < 0 0
History list length 24
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0 0, not started, process no 590, OS thread id 140366628529920
MySQL thread id 703864, query id 20047015 localhost root
SHOW ENGINE INNODB STATUS
---TRANSACTION 0 46989894, not started, process no 590, OS thread id 140366636758784
MySQL thread id 702822, query id 20046897 10.179.1.63 root
---TRANSACTION 0 46986223, ACTIVE 39782 sec, process no 590, OS thread id 140366626322176
25 lock struct(s), heap size 3024, 710 row lock(s), undo log entries 9
MySQL thread id 677706, query id 19994505 10.179.103.114 root
Trx read view will not see trx with id >= 0 46986224, sees < 0 46986224
Any further writes to the table then time out on me until I restart my MySQL server (or manually kill the deadlocked jobs):
PHP Fatal error: Lock wait timeout exceeded; try restarting transaction(Query: "UPDATE job_queue SET worker='108.166.81.250', attempts=attempts+1, started='2014-01-14 21:27:45', token='369eae55a7f0eacad3b678a3410de8e4' WHERE completed='0000-00-00 00:00:00' AND (started='0000-00-00 00:00:00' OR started<'2014-01-14 21:22:45') AND attempts<2 ORDER BY priority DESC, inserted LIMIT 1") in /utilities/Database.php on line 53
Can anyone explain to me why this query causes a deadlock? I was under the impression that all queries on an InnoDB table happened atomically. Any ideas?
This is causing a deadlock because the UPDATE query is locking all rows in the table, and depending on the indexes used (or lack thereof), two different sessions will potentially lock them in slightly different order. Remember that UPDATE, DELETE, and SELECT ... FOR UPDATE will lock all rows they encounter, whether those rows match all WHERE conditions or not. So when using them, you should try hard to be sure that they encounter as few rows as possible, by using indexes (ideally the primary key) and avoiding vague or wide-selecting conditions.
My suggestion for work queues is pretty much universal: Lock as little as possible as rarely as possible and always in a deterministic order. So, generally:
Use non-locking reads (regular SELECT) to find work to do by looking for stuff your worker knows how to do and is currently unclaimed (lease_owner IS NULL AND lease_expiry IS NULL -- or similar).
Choose one work item (or a few if you dare, but one is far simpler and normally allows for perfectly acceptable performance).
Update your work item (to claim it, but for any case it needs updating as well):
Open a transaction.
Lock your chosen work item with SELECT ... FOR UPDATE -- If it is no longer unclaimed, abort and choose another.
Update your chosen work item with your worker id and an expiry time for your lease on it.
Commit your transaction immediately.
Begin work on your leased work items.
In some other process, another poller looks for abandoned work and un-claims it (via the same update process above).
You can easily get very high throughput with this design (thousands of jobs per second), and with essentially no contention and no ordering issues. Optimizations to choose work that is less likely to conflict with other pollers are simple and effective (e.g. modulus on Job ID or similar, chosen to avoid starvation of jobs). The key is to remember that conflict on job selection is okay -- just abort and try again and everything moves along very quickly.
All locking writes for work queue items/jobs should be done only on single rows and by primary key only.

Deadlock with identical queries

I'm trying to resolve an error involving deadlocks on one of our busy tables. I've read this SO question about deadlocks and while it makes sense, the query order doesn't seem to be the cause in my case.
Here's the abbreviated output of SHOW ENGINE INNODB STATUS;:
*** (1) TRANSACTION:
TRANSACTION 1 2611184895, ACTIVE 0 sec, process no 17501, OS thread id 140516779579136 starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 368, 1 row lock(s)
MySQL thread id 211935717, query id 3146186174 [SERVER A] Searching rows for update
UPDATE images_unread_comments
SET unread = 0
WHERE user_id = 1 AND comment_id IN(1,2,3) AND unread = 1
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 404976 n bits 632 index `users_unread_comments` of table images_unread_comments trx id 1 2611184895 lock_mode X waiting
Record lock, heap no 558 PHYSICAL RECORD: n_fields 3; compact format; info bits 32
0: len 4; hex 0001461a; asc F ;; 1: len 1; hex 01; asc ;; 2: len 6; hex 00000e67d888; asc g ;;
*** (2) TRANSACTION:
TRANSACTION 1 2611184892, ACTIVE 0 sec, process no 17501, OS thread id 140516774520576 updating or deleting, thread declared inside InnoDB 494
mysql tables in use 1, locked 1
6 lock struct(s), heap size 1216, 11 row lock(s), undo log entries 1
MySQL thread id 211935715, query id 3146186169 [SERVER B] Updating
UPDATE images_unread_comments
SET unread = 0
WHERE user_id = 1 AND comment_id IN(1,2,3) AND unread = 1
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 404976 n bits 632 index users_unread_comments of table images_unread_comments trx id 1 2611184892 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
Record lock, heap no 555 PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 4; hex 0001461a; asc F ;; 1: len 1; hex 01; asc ;; 2: len 6; hex 00000e67daf0; asc g ;;
Record lock, heap no 556 PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 4; hex 0001461a; asc F ;; 1: len 1; hex 01; asc ;; 2: len 6; hex 00000e67dadb; asc g ;;
Record lock, heap no 557 PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 4; hex 0001461a; asc F ;; 1: len 1; hex 01; asc ;; 2: len 6; hex 00000e67d940; asc g #;;
Record lock, heap no 558 PHYSICAL RECORD: n_fields 3; compact format; info bits 32
0: len 4; hex 0001461a; asc F ;; 1: len 1; hex 01; asc ;; 2: len 6; hex 00000e67d888; asc g ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 404976 n bits 632 index users_unread_comments of table images_unread_comments trx id 1 2611184892 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 558 PHYSICAL RECORD: n_fields 3; compact format; info bits 32
0: len 4; hex 0001461a; asc F ;; 1: len 1; hex 01; asc ;; 2: len 6; hex 00000e67d888; asc g ;;
*** WE ROLL BACK TRANSACTION (1)
The thing I noticed is that the two SQL statements are identical; however one is being executed on Server A and the other on Server B. Regardless of why that is happening - why would this create a deadlock if both queries lock the same keys in the same order? Or am I misunderstanding the casue of deadlocks in the first place?
It seems that transaction 1 has performed another operation (insert?) in which it locked a gap in the index. It than waits for transaction 2 to perform the update, since 2 has locked the record with ID 1. Transaction 2 however can not proceed because transaction 1 holds a lock on the index. If you can isolate all SQL statements that are used in a transaction by this operation, we can see the exact reason for the deadlock

Mysql transaction waiting for lock which is already granted .. This is causing deadlock

If following situation a bug in mysql?.
Mysql Version: mysql.x86_64 5.0.77-4.el5_4.1
Kernel: Linux box2 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
------------------------
LATEST DETECTED DEADLOCK
------------------------
100125 4:24:41
*** (1) TRANSACTION:
TRANSACTION 0 210510625, ACTIVE 155 sec, process no 28125, OS thread id 1243162944 starting index read
mysql tables in use 1, locked 1
LOCK WAIT 5 lock struct(s), heap size 1216, undo log entries 1
MySQL thread id 162928579, query id 527252744 box22 172.16.11.105 user updating
delete from user_grid_items where user_id = 669786974 and START_X = 45 and START_Y = 65
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 61372 n bits 328 index `PRIMARY` of table `gamesutra_beta/user_grid_items` trx id 0 210510625 lock_mode X locks rec but not gap waiting
Record lock, heap no 127 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 8; hex 0000000027ec235e; asc ' #^;; 1: len 4; hex 0000002d; asc -;; 2: len 4; hex 00000041; asc A;; 3: len 6; hex 00000b561243; asc V C;; 4: len 7; hex 80000040070110; asc # ;; 5: len 23; hex 474949445f414e494d414c535f53515549445f50494e4b; asc GIID_ANIMALS_SQUID_PINK;; 6: len 4; hex cb59f060; asc Y `;; 7: len 4; hex 4b59f060; asc KY `;; 8: len 4; hex 80000000; asc ;; 9: len 1; hex 80; asc ;;
*** (2) TRANSACTION:
TRANSACTION 0 210505911, ACTIVE 555 sec, process no 28125, OS thread id 1184323904 starting index read, thread declared inside InnoDB 500
mysql tables in use 1, locked 1
5 lock struct(s), heap size 1216, undo log entries 1
MySQL thread id 162924258, query id 527252762 box22 172.16.11.105 user updating
delete from user_grid_items where user_id = 669786974 and START_X = 45 and START_Y = 65
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 61372 n bits 328 index `PRIMARY` of table `gamesutra_beta/user_grid_items` trx id 0 210505911 lock mode S locks rec but not gap
Record lock, heap no 127 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 8; hex 0000000027ec235e; asc ' #^;; 1: len 4; hex 0000002d; asc -;; 2: len 4; hex 00000041; asc A;; 3: len 6; hex 00000b561243; asc V C;; 4: len 7; hex 80000040070110; asc # ;; 5: len 23; hex 474949445f414e494d414c535f53515549445f50494e4b; asc GIID_ANIMALS_SQUID_PINK;; 6: len 4; hex cb59f060; asc Y `;; 7: len 4; hex 4b59f060; asc KY `;; 8: len 4; hex 80000000; asc ;; 9: len 1; hex 80; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 61372 n bits 328 index `PRIMARY` of table `gamesutra_beta/user_grid_items` trx id 0 210505911 lock_mode X locks rec but not gap waiting
Record lock, heap no 127 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 8; hex 0000000027ec235e; asc ' #^;; 1: len 4; hex 0000002d; asc -;; 2: len 4; hex 00000041; asc A;; 3: len 6; hex 00000b561243; asc V C;; 4: len 7; hex 80000040070110; asc # ;; 5: len 23; hex 474949445f414e494d414c535f53515549445f50494e4b; asc GIID_ANIMALS_SQUID_PINK;; 6: len 4; hex cb59f060; asc Y `;; 7: len 4; hex 4b59f060; asc KY `;; 8: len 4; hex 80000000; asc ;; 9: len 1; hex 80; asc ;;
*** WE ROLL BACK TRANSACTION (2)
------------
Sometimes the SHOW ENGINE INNODB STATUS can be hard to decipher because it only shows the current statement in the transaction. It does not show statements that occurred previously in the same transaction that may have acquired the locks that are actually being held.
In your case, a previous statement in transaction 2 acquired a shared lock on the row in question.
Then, transaction 1 attempted to acquire an exclusive lock on the same row, and is happily waiting for the shared lock to be removed.
Then, transaction 2, in another statement, attempted to acquire an exclusive lock on the same row. Classic deadlock. Neither transaction can finish.
One solution to help avoid such a deadlock would be to grab an exclusive lock on the row in transaction 2 with a SELECT FOR UPDATE statement, prior to the statement that's acquiring the shared lock.
I read something long ago, and not sure if it MIGHT be what you are running into as a result or not... without seeing code of the transaction of the current vs what it is conflicting with.
When processing your transactions, you should try to have them always do any locking in the same sequence... For an order / order detail / payments system, do the activities in the order listed as example here for all similar. If you have another process that tries its transaction in order of "Order detail / order", that CAN cause a deadlock.
One transaction is locking the order # first, then working the order detail. The other transaction locking the order detail first, then trying to get the order header.
HTH
Use SHOW ENGINE INNODB STATUS to determine the cause of the latest deadlock. That can help you to tune your application to avoid deadlocks.
Always be prepared to re-issue a transaction if it fails due to deadlock. Deadlocks are not dangerous. Just try again.