I have a Rails application running on mysql 5.6. 1-2 times a day the application hangs and restarting the server gets things back running.
In my logs, it looks like I get LOCK WAIT TIMEOUTS, The problem looks like it is in MySQL, in a SHOW ENGINE INNODB STATUS\G this is usually what we see:
---TRANSACTION 13339315173, ACTIVE 1339 sec
17 lock struct(s), heap size 3112, 28 row lock(s), undo log entries 236
MySQL thread id 28, OS thread handle 0x1a68, query id 37763097 localhost 127.0.0
.1 root cleaning up
Trx read view will not see trx with id >= 13339315174, sees < 13339315174
All other pending queries have the query strings in the status output, but this process that I believe is causing the hang, just shows this information.
If I wait a few moments and run the status again, I see the same type of transaction with some slight differences
---TRANSACTION 13339315173, ACTIVE 1572 sec
17 lock struct(s), heap size 3112, 28 row lock(s), undo log entries 265
MySQL thread id 28, OS thread handle 0x1a68, query id 38154939 localhost 127.0.0
.1 root cleaning up
Trx read view will not see trx with id >= 13339315174, sees < 13339315174
--------
I'm not very experienced with MySQL, but I'm inferring from this output that transaction 13339315173 is holding everything up. Can anyone shed some light on what may be going on?
My Rails application does not use transaction blocks, and the database itself is fairly small - less than 200 mb.
From some profiling tools, I have about 2,000 reads/sec and 30 writes/sec, with an average of 40-50 connections.
Full show innodb status below:
mysql> show engine innodb status \G
*************************** 1. row ***************************
Type: InnoDB
Name:
Status:
=====================================
2013-10-15 06:09:53 478 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 50 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 26859 srv_active, 0 srv_shutdown, 588 srv_idle
srv_master_thread log flush and writes: 27447
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 13486
OS WAIT ARRAY INFO: signal count 300223
Mutex spin waits 546801, rounds 492861, OS waits 10564
RW-shared spins 84659, rounds 148173, OS waits 2071
RW-excl spins 4010, rounds 320222, OS waits 663
Spin rounds per wait: 0.90 mutex, 1.75 RW-shared, 79.86 RW-excl
------------
TRANSACTIONS
------------
Trx id counter 13342016110
Purge done for trx's n:o < 13339316411 undo n:o < 0 state: running but idle
History list length 16262
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 71, OS thread handle 0x478, query id 38154940 localhost ::1 root
init
show engine innodb status
---TRANSACTION 13341449747, not started
MySQL thread id 68, OS thread handle 0x16d4, query id 38052625 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13341994684, not started
MySQL thread id 67, OS thread handle 0x870, query id 38131995 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13337736707, not started
MySQL thread id 66, OS thread handle 0x52c, query id 33240858 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13341868667, not started
MySQL thread id 63, OS thread handle 0x1f50, query id 37985312 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342003108, not started
MySQL thread id 62, OS thread handle 0xd64, query id 38140995 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13342015950, not started
MySQL thread id 60, OS thread handle 0x104c, query id 38154776 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342016109, not started
MySQL thread id 59, OS thread handle 0x2338, query id 38154937 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342015945, not started
MySQL thread id 58, OS thread handle 0x13f0, query id 38154767 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342015905, not started
MySQL thread id 57, OS thread handle 0x1758, query id 38154727 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13341953997, not started
MySQL thread id 56, OS thread handle 0x1e24, query id 38088539 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13341746417, not started
MySQL thread id 43, OS thread handle 0xfd8, query id 37843480 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311372513, not started
MySQL thread id 44, OS thread handle 0x9d0, query id 3625660 localhost 127.0.0.1
root cleaning up
---TRANSACTION 13341970398, not started
MySQL thread id 32, OS thread handle 0x1980, query id 38106072 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13311367975, not started
MySQL thread id 47, OS thread handle 0x2218, query id 3620225 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311372514, not started
MySQL thread id 45, OS thread handle 0x1c40, query id 3625666 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13342002623, not started
MySQL thread id 31, OS thread handle 0x192c, query id 38140455 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13311367962, not started
MySQL thread id 55, OS thread handle 0x1974, query id 3620144 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311367957, not started
MySQL thread id 54, OS thread handle 0x1aa4, query id 3620139 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311367952, not started
MySQL thread id 52, OS thread handle 0x2200, query id 3620116 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311367948, not started
MySQL thread id 51, OS thread handle 0x101c, query id 3620106 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311367998, not started
MySQL thread id 49, OS thread handle 0x23b4, query id 3620296 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311367946, not started
MySQL thread id 48, OS thread handle 0xbf8, query id 3620098 localhost 127.0.0.1
root cleaning up
---TRANSACTION 13311372515, not started
MySQL thread id 46, OS thread handle 0x2134, query id 3625651 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13341963034, not started
MySQL thread id 42, OS thread handle 0x1730, query id 38098217 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13341962985, not started
MySQL thread id 41, OS thread handle 0x2350, query id 38098156 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342004558, not started
MySQL thread id 30, OS thread handle 0x1f0c, query id 38142582 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342014064, not started
MySQL thread id 29, OS thread handle 0x1208, query id 38152727 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342010470, not started
MySQL thread id 26, OS thread handle 0xab4, query id 38148942 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311363353, not started
MySQL thread id 25, OS thread handle 0x1a38, query id 3614908 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13342015247, not started
MySQL thread id 24, OS thread handle 0x11ac, query id 38154020 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13341673205, ACTIVE 235 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 39, OS thread handle 0x17b8, query id 37762645 localhost 127.0.0
.1 root updating
UPDATE `stafftrack_logins` SET `end_time` = '2013-10-15 06:05:58', `updated_at`
= '2013-10-15 06:05:58' WHERE `id` = 140428
------- TRX HAS BEEN WAITING 235 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 245 page no 1355 n bits 216 index `PRIMARY` of table `rail
pro_development`.`stafftrack_logins` trx id 13341673205 lock_mode X locks rec bu
t not gap waiting
Record lock, heap no 136 PHYSICAL RECORD: n_fields 12; compact format; info bits
0
0: len 4; hex 8002248c; asc $ ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c2877; asc b (w;;
3: len 4; hex 80000011; asc ;;
4: len 4; hex 80000143; asc C;;
5: len 5; hex 9990de3e64; asc >d;;
6: len 5; hex 9990de6115; asc a ;;
7: len 1; hex 80; asc ;;
8: len 5; hex 9990de3e64; asc >d;;
9: len 5; hex 9990de6115; asc a ;;
10: SQL NULL;
11: SQL NULL;
------------------
---TRANSACTION 13341672862, ACTIVE 235 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 40, OS thread handle 0x20a8, query id 37762284 localhost 127.0.0
.1 root updating
UPDATE `rails` SET `stopped` = 1, `updated_at` = '2013-10-15 06:05:58' WHERE `id
` = 158
------- TRX HAS BEEN WAITING 235 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 225 page no 5 n bits 200 index `PRIMARY` of table `railpro
_development`.`rails` trx id 13341672862 lock_mode X locks rec but not gap waiti
ng
Record lock, heap no 95 PHYSICAL RECORD: n_fields 38; compact format; info bits
0
0: len 4; hex 8000009e; asc ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c22f6; asc b " ;;
3: len 4; hex 8000009e; asc ;;
4: len 15; hex 436c65616e20507265205374672038; asc Clean Pre Stg 8;;
5: len 1; hex 81; asc ;;
6: len 4; hex 80000005; asc ;;
7: len 4; hex 80000002; asc ;;
8: len 5; hex 998a973df8; asc = ;;
9: len 5; hex 9990de5b72; asc [r;;
10: len 4; hex 800224d9; asc $ ;;
11: len 4; hex 80000002; asc ;;
12: len 4; hex 8000005a; asc Z;;
13: len 1; hex 80; asc ;;
14: SQL NULL;
15: SQL NULL;
16: SQL NULL;
17: SQL NULL;
18: len 0; hex ; asc ;;
19: len 4; hex 43505338; asc CPS8;;
20: len 1; hex 81; asc ;;
21: len 4; hex 80000000; asc ;;
22: len 4; hex 80000000; asc ;;
23: SQL NULL;
24: len 1; hex 80; asc ;;
25: len 0; hex ; asc ;;
26: SQL NULL;
27: len 4; hex 800003e7; asc ;;
28: len 4; hex 800003e7; asc ;;
29: len 4; hex 80000002; asc ;;
30: len 4; hex 80000001; asc ;;
31: SQL NULL;
32: SQL NULL;
33: SQL NULL;
34: len 4; hex 80000005; asc ;;
35: len 1; hex 80; asc ;;
36: len 1; hex 80; asc ;;
37: len 1; hex 80; asc ;;
------------------
---TRANSACTION 13341581093, ACTIVE 300 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 69, OS thread handle 0x1f08, query id 37654138 localhost 127.0.0
.1 root updating
UPDATE `slings` SET `updated_at` = '2013-10-15 06:04:53', `tag` = 173 WHERE `id`
= 3760434
------- TRX HAS BEEN WAITING 300 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 237 page no 117 n bits 384 index `PRIMARY` of table `railp
ro_development`.`slings` trx id 13341581093 lock_mode X locks rec but not gap wa
iting
Record lock, heap no 138 PHYSICAL RECORD: n_fields 13; compact format; info bits
0
0: len 4; hex 80396132; asc 9a2;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c26e6; asc b & ;;
3: len 4; hex 800000dc; asc ;;
4: len 5; hex 9990de59d8; asc Y ;;
5: len 5; hex 9990de6073; asc `s;;
6: SQL NULL;
7: len 4; hex 80000009; asc ;;
8: len 4; hex 80000017; asc ;;
9: len 4; hex 800000b7; asc ;;
10: len 5; hex 9990de6073; asc `s;;
11: len 1; hex 30; asc 0;;
12: SQL NULL;
------------------
---TRANSACTION 13341529477, ACTIVE 332 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 38, OS thread handle 0x1fb0, query id 37598517 localhost 127.0.0
.1 root updating
UPDATE `stafftrack_logins` SET `end_time` = '2013-10-15 06:04:21', `updated_at`
= '2013-10-15 06:04:21' WHERE `id` = 140428
------- TRX HAS BEEN WAITING 332 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 245 page no 1355 n bits 208 index `PRIMARY` of table `rail
pro_development`.`stafftrack_logins` trx id 13341529477 lock_mode X locks rec bu
t not gap waiting
Record lock, heap no 136 PHYSICAL RECORD: n_fields 12; compact format; info bits
0
0: len 4; hex 8002248c; asc $ ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c2877; asc b (w;;
3: len 4; hex 80000011; asc ;;
4: len 4; hex 80000143; asc C;;
5: len 5; hex 9990de3e64; asc >d;;
6: len 5; hex 9990de6115; asc a ;;
7: len 1; hex 80; asc ;;
8: len 5; hex 9990de3e64; asc >d;;
9: len 5; hex 9990de6115; asc a ;;
10: SQL NULL;
11: SQL NULL;
------------------
---TRANSACTION 13341500243, ACTIVE 349 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 37, OS thread handle 0x1624, query id 37565181 localhost 127.0.0
.1 root updating
UPDATE `stafftrack_logins` SET `end_time` = '2013-10-15 06:04:04', `updated_at`
= '2013-10-15 06:04:04' WHERE `id` = 140421
------- TRX HAS BEEN WAITING 349 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 245 page no 1355 n bits 200 index `PRIMARY` of table `rail
pro_development`.`stafftrack_logins` trx id 13341500243 lock_mode X locks rec bu
t not gap waiting
Record lock, heap no 134 PHYSICAL RECORD: n_fields 12; compact format; info bits
0
0: len 4; hex 80022485; asc $ ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c2823; asc b (#;;
3: len 4; hex 8000003e; asc >;;
4: len 4; hex 800000d3; asc ;;
5: len 5; hex 9990de37d9; asc 7 ;;
6: len 5; hex 9990de6103; asc a ;;
7: len 1; hex 80; asc ;;
8: len 5; hex 9990de37d9; asc 7 ;;
9: len 5; hex 9990de6103; asc a ;;
10: SQL NULL;
11: SQL NULL;
------------------
---TRANSACTION 13341499101, ACTIVE 350 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 36, OS thread handle 0x19a8, query id 37563945 localhost 127.0.0
.1 root updating
UPDATE `stafftrack_logins` SET `end_time` = '2013-10-15 06:04:03', `updated_at`
= '2013-10-15 06:04:03' WHERE `id` = 140421
------- TRX HAS BEEN WAITING 350 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 245 page no 1355 n bits 200 index `PRIMARY` of table `rail
pro_development`.`stafftrack_logins` trx id 13341499101 lock_mode X locks rec bu
t not gap waiting
Record lock, heap no 134 PHYSICAL RECORD: n_fields 12; compact format; info bits
0
0: len 4; hex 80022485; asc $ ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c2823; asc b (#;;
3: len 4; hex 8000003e; asc >;;
4: len 4; hex 800000d3; asc ;;
5: len 5; hex 9990de37d9; asc 7 ;;
6: len 5; hex 9990de6103; asc a ;;
7: len 1; hex 80; asc ;;
8: len 5; hex 9990de37d9; asc 7 ;;
9: len 5; hex 9990de6103; asc a ;;
10: SQL NULL;
11: SQL NULL;
------------------
---TRANSACTION 13341486252, ACTIVE 360 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 35, OS thread handle 0x1538, query id 37550281 localhost 127.0.0
.1 root updating
UPDATE `stafftrack_logins` SET `end_time` = '2013-10-15 06:03:53', `updated_at`
= '2013-10-15 06:03:53' WHERE `id` = 140433
------- TRX HAS BEEN WAITING 360 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 245 page no 1355 n bits 200 index `PRIMARY` of table `rail
pro_development`.`stafftrack_logins` trx id 13341486252 lock_mode X locks rec bu
t not gap waiting
Record lock, heap no 133 PHYSICAL RECORD: n_fields 12; compact format; info bits
0
0: len 4; hex 80022491; asc $ ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c27f9; asc b ' ;;
3: len 4; hex 8000003e; asc >;;
4: len 4; hex 80000079; asc y;;
5: len 5; hex 9990de3eb3; asc > ;;
6: len 5; hex 9990de60ed; asc ` ;;
7: len 1; hex 80; asc ;;
8: len 5; hex 9990de3eb3; asc > ;;
9: len 5; hex 9990de60ed; asc ` ;;
10: SQL NULL;
11: SQL NULL;
------------------
---TRANSACTION 13341476145, ACTIVE 368 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 27, OS thread handle 0x1794, query id 37539510 localhost 127.0.0
.1 root updating
UPDATE `stafftrack_logins` SET `end_time` = '2013-10-15 06:03:45', `updated_at`
= '2013-10-15 06:03:45' WHERE `id` = 140433
------- TRX HAS BEEN WAITING 368 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 245 page no 1355 n bits 200 index `PRIMARY` of table `rail
pro_development`.`stafftrack_logins` trx id 13341476145 lock_mode X locks rec bu
t not gap waiting
Record lock, heap no 133 PHYSICAL RECORD: n_fields 12; compact format; info bits
0
0: len 4; hex 80022491; asc $ ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c27f9; asc b ' ;;
3: len 4; hex 8000003e; asc >;;
4: len 4; hex 80000079; asc y;;
5: len 5; hex 9990de3eb3; asc > ;;
6: len 5; hex 9990de60ed; asc ` ;;
7: len 1; hex 80; asc ;;
8: len 5; hex 9990de3eb3; asc > ;;
9: len 5; hex 9990de60ed; asc ` ;;
10: SQL NULL;
11: SQL NULL;
------------------
---TRANSACTION 13341210786, ACTIVE 561 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 61, OS thread handle 0xcc0, query id 37228004 localhost 127.0.0.
1 root updating
UPDATE `slings` SET `tag` = 221, `updated_at` = '2013-10-15 06:00:32' WHERE `id`
= 3760256
------- TRX HAS BEEN WAITING 561 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 237 page no 113 n bits 408 index `PRIMARY` of table `railp
ro_development`.`slings` trx id 13341210786 lock_mode X locks rec but not gap wa
iting
Record lock, heap no 240 PHYSICAL RECORD: n_fields 13; compact format; info bits
0
0: len 4; hex 80396080; asc 9` ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c2365; asc b #e;;
3: len 4; hex 800000dc; asc ;;
4: len 5; hex 9990de5644; asc VD;;
5: len 5; hex 9990de5c11; asc \ ;;
6: SQL NULL;
7: len 4; hex 80000008; asc ;;
8: len 4; hex 8000001f; asc ;;
9: len 4; hex 800000de; asc ;;
10: len 5; hex 9990de5c11; asc \ ;;
11: len 1; hex 30; asc 0;;
12: SQL NULL;
------------------
---TRANSACTION 13340733149, ACTIVE 813 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 34, OS thread handle 0x1c74, query id 36686947 localhost 127.0.0
.1 root updating
UPDATE `slings` SET `tag` = 0, `updated_at` = '2013-10-15 05:56:20' WHERE `id` =
3750500
------- TRX HAS BEEN WAITING 813 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 237 page no 103 n bits 368 index `PRIMARY` of table `railp
ro_development`.`slings` trx id 13340733149 lock_mode X locks rec but not gap wa
iting
Record lock, heap no 299 PHYSICAL RECORD: n_fields 13; compact format; info bits
0
0: len 4; hex 80393a64; asc 9:d;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c258d; asc b % ;;
3: len 4; hex 800000dc; asc ;;
4: len 5; hex 9990d6ad21; asc !;;
5: len 5; hex 9990de5e09; asc ^ ;;
6: SQL NULL;
7: len 4; hex 80000007; asc ;;
8: len 4; hex 8000001e; asc ;;
9: len 4; hex 800000d3; asc ;;
10: len 5; hex 9990de5e09; asc ^ ;;
11: len 1; hex 30; asc 0;;
12: SQL NULL;
------------------
---TRANSACTION 13340568617, ACTIVE 903 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 33, OS thread handle 0xfd4, query id 36497676 localhost 127.0.0.
1 root updating
UPDATE `slings` SET `tag` = 183, `updated_at` = '2013-10-15 05:54:50' WHERE `id`
= 3760434
------- TRX HAS BEEN WAITING 903 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 237 page no 117 n bits 328 index `PRIMARY` of table `railp
ro_development`.`slings` trx id 13340568617 lock_mode X locks rec but not gap wa
iting
Record lock, heap no 138 PHYSICAL RECORD: n_fields 13; compact format; info bits
0
0: len 4; hex 80396132; asc 9a2;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c26e6; asc b & ;;
3: len 4; hex 800000dc; asc ;;
4: len 5; hex 9990de59d8; asc Y ;;
5: len 5; hex 9990de6073; asc `s;;
6: SQL NULL;
7: len 4; hex 80000009; asc ;;
8: len 4; hex 80000017; asc ;;
9: len 4; hex 800000b7; asc ;;
10: len 5; hex 9990de6073; asc `s;;
11: len 1; hex 30; asc 0;;
12: SQL NULL;
------------------
---TRANSACTION 13339315173, ACTIVE 1572 sec
17 lock struct(s), heap size 3112, 28 row lock(s), undo log entries 265
MySQL thread id 28, OS thread handle 0x1a68, query id 38154939 localhost 127.0.0
.1 root cleaning up
Trx read view will not see trx with id >= 13339315174, sees < 13339315174
--------
FILE I/O
--------
I/O thread 0 state: wait Windows aio (insert buffer thread)
I/O thread 1 state: wait Windows aio (log thread)
I/O thread 2 state: wait Windows aio (read thread)
I/O thread 3 state: wait Windows aio (read thread)
I/O thread 4 state: wait Windows aio (read thread)
I/O thread 5 state: wait Windows aio (read thread)
I/O thread 6 state: wait Windows aio (write thread)
I/O thread 7 state: wait Windows aio (write thread)
I/O thread 8 state: wait Windows aio (write thread)
I/O thread 9 state: wait Windows aio (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
16009 OS file reads, 247119 OS file writes, 196868 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 4.14 writes/s, 4.14 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 600, seg size 602, 461 merges
merged operations:
insert 234, delete mark 72731, delete 6105
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 25499819, node heap has 1457 buffer(s)
18360.79 hash searches/s, 6378.53 non-hash searches/s
---
LOG
---
Log sequence number 32913782367
Log flushed up to 32913782357
Pages flushed up to 32905803711
Last checkpoint at 32905803711
0 pending log writes, 0 pending chkp writes
154012 log i/o's done, 4.14 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 13186891776; in additional pool allocated 0
Dictionary memory allocated 704979
Buffer pool size 786432
Free buffers 768193
Database pages 16782
Old database pages 6349
Modified db pages 1585
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 177, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 15892, created 890, written 74469
0.00 reads/s, 0.02 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 16782, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 98304
Free buffers 95915
Database pages 2211
Old database pages 836
Modified db pages 191
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 34, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2119, created 92, written 12239
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2211, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 98304
Free buffers 96184
Database pages 1931
Old database pages 732
Modified db pages 234
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 15, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1831, created 100, written 5972
0.00 reads/s, 0.02 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 1931, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 98304
Free buffers 96026
Database pages 2098
Old database pages 794
Modified db pages 104
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 28, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2090, created 8, written 2450
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2098, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size 98304
Free buffers 95810
Database pages 2311
Old database pages 872
Modified db pages 354
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 11, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2012, created 299, written 20722
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2311, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size 98304
Free buffers 96204
Database pages 1915
Old database pages 726
Modified db pages 128
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 24, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1850, created 65, written 10904
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 1915, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size 98304
Free buffers 95833
Database pages 2288
Old database pages 864
Modified db pages 196
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 20, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2201, created 87, written 6023
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2288, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size 98304
Free buffers 95952
Database pages 2173
Old database pages 821
Modified db pages 224
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 23, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2079, created 94, written 10846
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2173, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size 98304
Free buffers 96269
Database pages 1855
Old database pages 704
Modified db pages 154
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 22, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1710, created 145, written 5313
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 1855, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
1 read views open inside InnoDB
Main thread id 7604, state: sleeping
Number of rows inserted 12203, updated 144283, deleted 45540, read 8849139354
0.36 inserts/s, 4.46 updates/s, 0.04 deletes/s, 119285.59 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
1 row in set (0.00 sec)
mysql>
In case anyone comes across this in the future. We found the problem was related to mutli-threading and a bug in Rails 2.3.2 handling connections.
The jist of it was that a connection was being released before we were actually done using it, so sometimes in the middle of a transaction the mysql thread was being passed to a different process, and getting itself in to nested transaction cascading locks throughout the application.
To fix this we updated to rails 2.3.18, and backported a lot of Rails 3 ActiveRecord.
Related
I have a question about the deadlock from optimistic lock.
I implemented an simple airline booking system. The database scheme like this:
In the book flight API, the method will check if the flight is available and get the version number. Then the method will minus 1 from available_seats and check if the version number is same. If not the save() method will throw ObjectOptimisticLockingFailureException and catch by controller layer. The code is the service layer:
public Ticket bookFlight(Flight flight, long customerId) throws Exception {
Flight returnedFlight = flightRepository.findFlightByFlightNumberAndFlightDate(flight.getFlightNumber(), flight.getFlightDate());
if (validFlightIsAvailable(returnedFlight) == false) {
// A valid flight ticket id should greater than 0.
// Therefore, ticketId = 0 means the flight is full.
int fullFlightTicketId = 0;
log.info("Customer " + customerId + " failed to book the ticket because the flight is full.");
return new Ticket(fullFlightTicketId);
} else {
Ticket newTicket = new Ticket(customerId, returnedFlight.getFlightId(), returnedFlight.getFlightDate());
if (checkIsDuplicatedBooking(newTicket) == true){
log.error("The customer (" + customerId + ") already book the ticket for flight " + newTicket.getFlightId());
throw new ClientException("Customer already booked the ticket in the same flight", HttpStatus.BAD_REQUEST);
}
returnedFlight.setAvailableSeats(returnedFlight.getAvailableSeats() - 1);
Flight newFlight = flightRepository.save(returnedFlight);
if (newFlight == null) {
log.error("Update flight's available seats error. (flight Id: " +
returnedFlight.getFlightId() + "). Rollback all transactions.");
throw new ServerException("Unknown Server Exception.", HttpStatus.INTERNAL_SERVER_ERROR);
}
Ticket returnedTicket = ticketRepository.save(newTicket);
if (returnedTicket != null) {
log.info("Customer Id: " + customerId + " successfully booked the ticket in flight " + flight.getFlightId());
return returnedTicket;
} else {
log.error("Save new ticket into database error. (Customer Id: " + customerId + ", flight Id: " +
returnedFlight.getFlightId() + "). Rollback all transactions.");
throw new ServerException("Unknown Server Exception.", HttpStatus.INTERNAL_SERVER_ERROR);
}
}
}
The flight entity is :
#Entity
#Table(name = "flight")
public class Flight {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
#Column(name = "flight_id", nullable = false)
private Long flightId;
#Column(name = "flight_number", nullable = false)
private Long flightNumber;
#Column(name = "flight_date", nullable = false)
private Date flightDate;
#Column(name = "available_seats", nullable = false)
private Integer availableSeats;
#Column(name = "version")
#Version
private int version;
// Getter and Setter
I added a unit test for 50 users and use mockMvc to test the function.
#Test
void bookFlight_Concurrent_Success() throws Exception {
// Get all flight by Default flight number
List<Flight> availableFlights = flightService.getAllAvailableFlightsByFlightNumber(defaultFlights.get(0));
int flightIndex = 3;
int totalAvailableSeat = availableFlights.get(flightIndex).getAvailableSeats();
String requestJSON = "{\n" +
" \"flightDate\": \"" + availableFlights.get(flightIndex).getFlightDate().toString() + "\",\n" +
" \"flightNumber\": " + availableFlights.get(flightIndex).getFlightNumber() + "\n" +
"}";
// Set up builders
List<RequestBuilder> builders = new ArrayList<>();
for (int i = 0; i < defaultCustomerUsernames.size(); i++){
String jwt = getJWTByUsername(defaultCustomerUsernames.get(i), constants.CUSTOMER_USER_PASSWORD_0);
RequestBuilder builder = post("/bookFlight").header("Authorization", "Bearer " + jwt).
accept(MediaType.APPLICATION_JSON).content(requestJSON).contentType(MediaType.APPLICATION_JSON);
builders.add(builder);
}
// Start threads to book flight
List<Thread> threadList = new ArrayList<>();
for (int i = 0; i < defaultCustomerUsernames.size(); i++){
int finalI0 = i;
int finalI1 = i;
Thread thread = new Thread(){
public void run() {
try {
System.out.println(this.getName() + ": Start thread " + finalI1);
MvcResult result = mockMvc.perform(builders.get(finalI0)).andReturn();
System.out.println(this.getName() + ": Finish thread " + finalI1);
} catch (Exception e) {
e.printStackTrace();
}
}
};
thread.start();
threadList.add(thread);
}
try{
for (Thread thread: threadList){
thread.join();
}
}
catch (InterruptedException e){
e.printStackTrace();
}
// Verify the left ticket + booked ticket is full amount
int bookedTicketNum = ticketRepository.findTicketsByFlightId(availableFlights.get(flightIndex).getFlightId()).size();
int newAvailableTicket = flightRepository.findFlightByFlightId(availableFlights.get(flightIndex).getFlightId()).getAvailableSeats();
System.out.println("Booked Ticket Num: " + bookedTicketNum);
System.out.println("New Available Ticket: " + newAvailableTicket);
assertEquals(totalAvailableSeat, bookedTicketNum+newAvailableTicket);
}
beforeAll and afterAll are implemented to generate the default flight and customer user.
The test passed but
2020-09-22 21:34:00:667 ERROR [Thread-7] org.hibernate.engine.jdbc.batch.internal.BatchingBatch [130] : HHH000315: Exception executing batch [java.sql.BatchUpdateException: Deadlock found when trying to get lock; try restarting transaction], SQL: update flight set available_seats=?, flight_date=?, flight_number=?, version=? where flight_id=? and version=?
2020-09-22 21:34:00:668 ERROR [Thread-7] org.hibernate.engine.jdbc.spi.SqlExceptionHelper [142] : Deadlock found when trying to get lock; try restarting transaction
I used SHOW ENGINE INNODB STATUS to check deadlock log and get this:
=====================================
2020-09-22 21:36:50 0x700001f89000 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 17 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 1595 srv_active, 0 srv_shutdown, 22112 srv_idle
srv_master_thread log flush and writes: 0
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 2965
OS WAIT ARRAY INFO: signal count 44793
RW-shared spins 164384, rounds 166427, OS waits 1899
RW-excl spins 63944, rounds 93238, OS waits 183
RW-sx spins 28152, rounds 88279, OS waits 316
Spin rounds per wait: 1.01 RW-shared, 1.46 RW-excl, 3.14 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2020-09-22 21:34:00 0x70000156e000
*** (1) TRANSACTION:
TRANSACTION 34961, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 1
MySQL thread id 586, OS thread handle 123145342349312, query id 3508200 localhost 127.0.0.1 root updating
update flight set available_seats=149, flight_date='2020-12-14', flight_number=2, version=7 where flight_id=178360 and version=6
*** (1) HOLDS THE LOCK(S):
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 34961 lock mode S locks rec but not gap
Record lock, heap no 85 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002b8b8; asc ;;
1: len 6; hex 000000008889; asc ;;
2: len 7; hex 02000002091880; asc ;;
3: len 4; hex 80000002; asc ;;
4: len 3; hex 8fc98e; asc ;;
5: len 4; hex 80000095; asc ;;
6: len 4; hex 80000007; asc ;;
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 34961 lock_mode X locks rec but not gap waiting
Record lock, heap no 85 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002b8b8; asc ;;
1: len 6; hex 000000008889; asc ;;
2: len 7; hex 02000002091880; asc ;;
3: len 4; hex 80000002; asc ;;
4: len 3; hex 8fc98e; asc ;;
5: len 4; hex 80000095; asc ;;
6: len 4; hex 80000007; asc ;;
*** (2) TRANSACTION:
TRANSACTION 34959, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 1
MySQL thread id 591, OS thread handle 123145335984128, query id 3508204 localhost 127.0.0.1 root updating
update flight set available_seats=149, flight_date='2020-12-14', flight_number=2, version=7 where flight_id=178360 and version=6
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 34959 lock mode S locks rec but not gap
Record lock, heap no 85 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002b8b8; asc ;;
1: len 6; hex 000000008889; asc ;;
2: len 7; hex 02000002091880; asc ;;
3: len 4; hex 80000002; asc ;;
4: len 3; hex 8fc98e; asc ;;
5: len 4; hex 80000095; asc ;;
6: len 4; hex 80000007; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 34959 lock_mode X locks rec but not gap waiting
Record lock, heap no 85 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002b8b8; asc ;;
1: len 6; hex 000000008889; asc ;;
2: len 7; hex 02000002091880; asc ;;
3: len 4; hex 80000002; asc ;;
4: len 3; hex 8fc98e; asc ;;
5: len 4; hex 80000095; asc ;;
6: len 4; hex 80000007; asc ;;
*** WE ROLL BACK TRANSACTION (2)
------------
TRANSACTIONS
------------
Trx id counter 35089
Purge done for trx's n:o < 35089 undo n:o < 0 state: running but idle
History list length 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 422038541902072, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541902912, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541901232, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541897872, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541899552, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541898712, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541897032, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541896192, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541895352, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (read thread)
I/O thread 4 state: waiting for i/o request (read thread)
I/O thread 5 state: waiting for i/o request (read thread)
I/O thread 6 state: waiting for i/o request (write thread)
I/O thread 7 state: waiting for i/o request (write thread)
I/O thread 8 state: waiting for i/o request (write thread)
I/O thread 9 state: waiting for i/o request (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 1
1276 OS file reads, 982454 OS file writes, 737797 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 2 buffer(s)
Hash table size 34679, node heap has 2 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 4 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 1793856320
Log buffer assigned up to 1793856320
Log buffer completed up to 1793856320
Log written up to 1793856320
Log flushed up to 1793856320
Added dirty pages up to 1793856320
Pages flushed up to 1793856320
Last checkpoint at 1793856320
937447 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 137363456
Dictionary memory allocated 476872
Buffer pool size 8191
Free buffers 5484
Database pages 2694
Old database pages 982
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 4, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1241, created 1453, written 34817
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2694, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=2483, Main thread ID=0x70000177a000 , state=sleeping
Number of rows inserted 2692764, updated 1168, deleted 1168363, read 1256843
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
Number of system rows inserted 22, updated 377, deleted 20, read 13121
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
I think two update transaction cause this issue.
I tried to manual debug, and still can use other tools update the row after Flight newFlight = flightRepository.save(returnedFlight);
But, I cannot update the row after Ticket returnedTicket = ticketRepository.save(newTicket);
The deadlock log:
=====================================
2020-09-22 21:41:23 0x700001f89000 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 29 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 1597 srv_active, 0 srv_shutdown, 22382 srv_idle
srv_master_thread log flush and writes: 0
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 2965
OS WAIT ARRAY INFO: signal count 44793
RW-shared spins 164384, rounds 166427, OS waits 1899
RW-excl spins 63944, rounds 93238, OS waits 183
RW-sx spins 28152, rounds 88279, OS waits 316
Spin rounds per wait: 1.01 RW-shared, 1.46 RW-excl, 3.14 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2020-09-22 21:41:17 0x70000156e000
*** (1) TRANSACTION:
TRANSACTION 35092, ACTIVE 24 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 1136, 1 row lock(s)
MySQL thread id 8, OS thread handle 123145333862400, query id 3508825 localhost 127.0.0.1 root updating
/* ApplicationName=DataGrip 2020.2.2 */ UPDATE airline_booking_system.flight t SET t.version = 12 WHERE t.flight_id = 173431
*** (1) HOLDS THE LOCK(S):
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 35092 lock_mode X locks rec but not gap waiting
Record lock, heap no 4 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002a577; asc w;;
1: len 6; hex 000000008911; asc ;;
2: len 7; hex 020000016c0c77; asc l w;;
3: len 4; hex 80000373; asc s;;
4: len 3; hex 8fc938; asc 8;;
5: len 4; hex 80000096; asc ;;
6: len 4; hex 8000000b; asc ;;
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 35092 lock_mode X locks rec but not gap waiting
Record lock, heap no 4 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002a577; asc w;;
1: len 6; hex 000000008911; asc ;;
2: len 7; hex 020000016c0c77; asc l w;;
3: len 4; hex 80000373; asc s;;
4: len 3; hex 8fc938; asc 8;;
5: len 4; hex 80000096; asc ;;
6: len 4; hex 8000000b; asc ;;
*** (2) TRANSACTION:
TRANSACTION 35091, ACTIVE 41 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 1
MySQL thread id 599, OS thread handle 123145336287232, query id 3508827 localhost 127.0.0.1 root updating
update flight set available_seats=149, flight_date='2020-09-24', flight_number=883, version=11 where flight_id=173431 and version=10
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 35091 lock mode S locks rec but not gap
Record lock, heap no 4 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002a577; asc w;;
1: len 6; hex 000000008911; asc ;;
2: len 7; hex 020000016c0c77; asc l w;;
3: len 4; hex 80000373; asc s;;
4: len 3; hex 8fc938; asc 8;;
5: len 4; hex 80000096; asc ;;
6: len 4; hex 8000000b; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 35091 lock_mode X locks rec but not gap waiting
Record lock, heap no 4 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002a577; asc w;;
1: len 6; hex 000000008911; asc ;;
2: len 7; hex 020000016c0c77; asc l w;;
3: len 4; hex 80000373; asc s;;
4: len 3; hex 8fc938; asc 8;;
5: len 4; hex 80000096; asc ;;
6: len 4; hex 8000000b; asc ;;
*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------
Trx id counter 35093
Purge done for trx's n:o < 35091 undo n:o < 0 state: running but idle
History list length 1
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 422038541900392, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541902072, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541902912, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541901232, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541897872, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541899552, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541898712, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541897032, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541896192, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541895352, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (read thread)
I/O thread 4 state: waiting for i/o request (read thread)
I/O thread 5 state: waiting for i/o request (read thread)
I/O thread 6 state: waiting for i/o request (write thread)
I/O thread 7 state: waiting for i/o request (write thread)
I/O thread 8 state: waiting for i/o request (write thread)
I/O thread 9 state: waiting for i/o request (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 1
1276 OS file reads, 982489 OS file writes, 737827 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.24 writes/s, 0.17 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 2 buffer(s)
Hash table size 34679, node heap has 2 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 4 buffer(s)
0.03 hash searches/s, 0.10 non-hash searches/s
---
LOG
---
Log sequence number 1793857089
Log buffer assigned up to 1793857089
Log buffer completed up to 1793857089
Log written up to 1793857089
Log flushed up to 1793857089
Added dirty pages up to 1793857089
Pages flushed up to 1793857089
Last checkpoint at 1793857089
937456 log i/o's done, 0.03 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 137363456
Dictionary memory allocated 476872
Buffer pool size 8191
Free buffers 5484
Database pages 2694
Old database pages 982
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 4, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1241, created 1453, written 34832
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2694, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=2483, Main thread ID=0x70000177a000 , state=sleeping
Number of rows inserted 2692765, updated 1169, deleted 1168363, read 1256855
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.03 reads/s
Number of system rows inserted 22, updated 377, deleted 20, read 13205
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
If anyone can help me? I tried Pessimistic Lock and no issue.
I have Mysql5.5 on windows system. Since past 1 month it is running slow.
Here is the query which are running the most:
begin
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1
where roomlogs.netId = pnetid
and roomlogs.`status` = 0
and roomlogs.enterTime>=DATE(ptime)
and roomlogs.enterTime<DATE(ptime) + interval 1 day;
INSERT
into roomlogs (roomlogs.roomId, roomlogs.netId, roomlogs.enterTime,
roomlogs.exitTime, roomlogs.hidden, roomlogs.checkinId)
value
(pRoomId, pnetid,ptime, pexit, phidden, pcheckid);
select *
from roomlogs
where roomlogs.roomlogId = LAST_INSERT_ID();
end
AND:
begin
select count(*) as total
from roomlogs
where roomId = proomId
and roomlogs.`status` = 0
AND roomlogs.enterTime >= CURDATE()
AND roomlogs.enterTime < CURDATE() + INTERVAL 1 DAY
and roomlogs.hidden!=1;
end
Here index of roomlogs are primary(roomlogId), NETID(netid,status,enterTime), ROOMID(roomID,status,enterTime)
Here is my InnoDB status when MySQL has more number of queries (around 100) running simultaneously in the server.
=====================================
2018-10-25 17:42:01 1728 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 20 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 148860 srv_active, 0 srv_shutdown, 271514 srv_idle
srv_master_thread log flush and writes: 420374
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 80056
OS WAIT ARRAY INFO: signal count 80039
Mutex spin waits 6920930, rounds 7674964, OS waits 52940
RW-shared spins 27042, rounds 803211, OS waits 26514
RW-excl spins 1173, rounds 9109, OS waits 211
Spin rounds per wait: 1.11 mutex, 29.70 RW-shared, 7.77 RW-excl
------------
TRANSACTIONS
------------
Trx id counter 78706084
Purge done for trx's n:o < 78706079 undo n:o < 0 state: running but idle
History list length 2281
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 78705887, not started
MySQL thread id 3000, OS thread handle 0x8e0, query id 4915274 localhost 127.0.0.1 root cleaning up
**------------ SKIPPING SOME DATA HERE--------------**
---TRANSACTION 78701492, not started
MySQL thread id 15, OS thread handle 0x12c4, query id 4889345 localhost 127.0.0.1 root cleaning up
---TRANSACTION 78697207, not started
MySQL thread id 16, OS thread handle 0x408, query id 4889123 localhost 127.0.0.1 root cleaning up
---TRANSACTION 78701491, not started
MySQL thread id 14, OS thread handle 0x11c8, query id 4889320 localhost 127.0.0.1 root cleaning up
---TRANSACTION 78697202, not started
MySQL thread id 13, OS thread handle 0x10dc, query id 4889098 localhost 127.0.0.1 root cleaning up
---TRANSACTION 78706073, ACTIVE 1 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2989, OS thread handle 0x157c, query id 4916082 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 1 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78706073 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78706056, ACTIVE 4 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2942, OS thread handle 0x15e0, query id 4916017 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 4 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78706056 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78706045, ACTIVE 5 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2976, OS thread handle 0x125c, query id 4915976 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 5 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78706045 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78706009, ACTIVE 9 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2946, OS thread handle 0xb98, query id 4915824 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 9 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78706009 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78706004, ACTIVE 10 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2995, OS thread handle 0x1234, query id 4915806 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 10 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78706004 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78705997, ACTIVE 12 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2948, OS thread handle 0x1708, query id 4915783 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 12 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78705997 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78705985, ACTIVE 14 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2986, OS thread handle 0x1640, query id 4915747 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 14 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78705985 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78705977, ACTIVE 16 sec fetching rows
mysql tables in use 1, locked 1
4610 lock struct(s), heap size 539064, 1001945 row lock(s)
MySQL thread id 2938, OS thread handle 0x10bc, query id 4915725 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
--------
FILE I/O
--------
I/O thread 0 state: wait Windows aio (insert buffer thread)
I/O thread 1 state: wait Windows aio (log thread)
I/O thread 2 state: wait Windows aio (read thread)
I/O thread 3 state: wait Windows aio (read thread)
I/O thread 4 state: wait Windows aio (read thread)
I/O thread 5 state: wait Windows aio (read thread)
I/O thread 6 state: wait Windows aio (write thread)
I/O thread 7 state: wait Windows aio (write thread)
I/O thread 8 state: wait Windows aio (write thread)
I/O thread 9 state: wait Windows aio (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
23467 OS file reads, 931766 OS file writes, 524775 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 1.00 writes/s, 1.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 39, seg size 41, 61 merges
merged operations:
insert 57, delete mark 5, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 4425293, node heap has 2003 buffer(s)
30.00 hash searches/s, 19.95 non-hash searches/s
---
LOG
---
Log sequence number 7396884835
Log flushed up to 7396884835
Pages flushed up to 7396717673
Last checkpoint at 7396717673
0 pending log writes, 0 pending chkp writes
164342 log i/o's done, 1.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 2197815296; in additional pool allocated 0
Dictionary memory allocated 2793588
Buffer pool size 131072
Free buffers 101935
Database pages 27102
Old database pages 9967
Modified db pages 377
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 498, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 22500, created 4602, written 613885
0.00 reads/s, 0.10 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 27102, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 16384
Free buffers 12818
Database pages 3312
Old database pages 1240
Modified db pages 63
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 72, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2713, created 599, written 81358
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3312, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 16384
Free buffers 12809
Database pages 3319
Old database pages 1215
Modified db pages 27
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 15, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2770, created 549, written 78718
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3319, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 16384
Free buffers 12717
Database pages 3411
Old database pages 1256
Modified db pages 28
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 77, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2795, created 616, written 49290
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3411, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size 16384
Free buffers 12733
Database pages 3395
Old database pages 1246
Modified db pages 51
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 45, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2780, created 615, written 91680
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3395, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size 16384
Free buffers 12812
Database pages 3318
Old database pages 1217
Modified db pages 47
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 108, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2645, created 673, written 95064
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3318, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size 16384
Free buffers 12590
Database pages 3538
Old database pages 1294
Modified db pages 51
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 35, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2905, created 633, written 89554
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3538, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size 16384
Free buffers 12563
Database pages 3567
Old database pages 1307
Modified db pages 52
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 32, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 3127, created 440, written 56311
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3567, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size 16384
Free buffers 12893
Database pages 3242
Old database pages 1192
Modified db pages 58
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 114, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2765, created 477, written 71910
0.00 reads/s, 0.10 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3242, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
1 read views open inside InnoDB
Main thread id 5080, state: sleeping
Number of rows inserted 97780, updated 862074, deleted 219, read 468159566176
1.55 inserts/s, 1.20 updates/s, 0.00 deletes/s, 3363648.47 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
So, Here you can see that transaction was waiting for a record lock, Also the last transaction with ID 78705977 is locking 1001945 rows which it should be as it is not updating that much rows.
Also Why it is taking time to fetch the rows like 16 seconds?
Here is create Table of roomlogs:
CREATE TABLE `roomlogs` (
`roomId` varchar(200) CHARACTER SET latin1 DEFAULT NULL,
`netId` varchar(50) CHARACTER SET latin1 DEFAULT NULL,
`enterTime` datetime DEFAULT NULL,
`exitTime` datetime DEFAULT NULL,
`roomlogId` int(11) NOT NULL AUTO_INCREMENT,
`status` int(11) DEFAULT '0',
`hidden` int(11) DEFAULT '0',
`checkinId` int(11) DEFAULT '-1',
PRIMARY KEY (`roomlogId`),
KEY `NETID` (`netId`,`status`,`enterTime`),
KEY `RoomLogIndex` (`roomId`,`status`,`enterTime`)
) ENGINE=InnoDB AUTO_INCREMENT=1315726 DEFAULT CHARSET=utf8
Normally the contention should not happen because of the status - attribute which is updated at the same time.
But because of MVCC the different transaction do not recognize that the records are not to be changed anymore. Therefore they all see the same records and lock them even if they would not have to do that because of the already changed status.
I suspect, that for you it is important, that the single change is done.
At the moment the transaction A (for example) locks all the records as they seem to be changed, according to the state of the data beginning of the transaction.
As soon as all records could be locked by A because all transactions running parallel at the beginning have been committed, transaction A locks and fetches those records, finds out, that the status has already been changed and does the changes on the leftovers.
After that change the originally locked set of records are unlocked.
The leftovers might not be many, but the contention is heavy.
The solution I see:
There seem to be many parallel threads changing the records with the same netid. Can that be avoided?
if not, I never tried that, but perhaps changing the isolation level can help: See: https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html
I don't (yet) see why the update is stumbling over itself. But I see some things that might speed up other things, thereby helping indirectly:
Item 1, faster select
select *
from roomlogs
where roomlogs.roomlogId = LAST_INSERT_ID();
Could (perhaps) be replaced by
SELECT LAST_INSERT_ID(),
pRoomId, pnetid, ptime, pexit, phidden, pcheckid;
Caution: I am assuming that those are the only columns, and that the order is correct. Please provide SHOW CREATE TABLE roomlogs so we don't have to guess.
Item 2, covering index
INDEX (roomID, status, enterTime)
-->
INDEX (roomID, status, enterTime, hidden)
This would be a "covering index", thereby speeding up the SELECT COUNT(*) ...
Item 3, better optimization
If hidden has only values 0 and 1, then
hidden != 1
-->
hidden = 0
The Optimizer can handle = better than !=. At that point, this is an even better index for item 2:
INDEX (roomID, status, hidden, enterTime)
Note: I did not recommend this in Item 2 because of the !=.
Item 4, datatype:
If status is not some form of INT, then I will have the answer. Again, need SHOW CREATE TABLE roomlogs
Something to try:
begin
START TRANSACTION;
SELECT *
FROM roomlogs
WHERE netId = pnetid
AND `status` = 0
AND enterTime >= DATE(ptime)
AND enterTime < DATE(ptime) + interval 1 day;
FOR UPDATE; -- This might cause different locking action
-- WHERE clause is equivalent to UPDATE's
-- The following are as before (or as further optimized):
UPDATE ...
INSERT ...
SELECT ...
COMMIT;
end;
A possible reason: In older versions of MySQL, a different, less sophisticated, Optimizer was used in UPDATE and DELETE. Hence, upgrading might be a solution.
Check for errors. You could get a timeout or deadlock. Either need to be handled.
(Yeah, I know this is my 3rd answer. But it is in a different direction.)
Consider moving status out of the table. This is especially since multiple rows need to be changed. Also since you have a datetime, but the status is associated with a date.
The new table would have
roomid ...
status TINYINT UNSIGNED NOT NULL (default 0?)
date DATE NOT NULL -- comes from DATE(enterTime)
PRIMARY KEY(roomid)
You would JOIN to the table when you need to see the status.
You would update the one row when you need to change the status. And you would not necessarily touch the other table.
This idea will probably fail if you need some rows to be status=0 and some status=1 simultaneously. However, if only 'older' rows have =0, then adding a time to the table would say "status is 0 before the time". This would make the JOIN a bit more complex, but still doable.
Please tell us what status "means". And consider changing the column name to reflect the semantics.
When I try to make UPDATE with subselect I catch deadlocks.
I caught this deadlock in my application I am using Hibernate for get data from DB and in my log, I see next:
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
When I removed subselect from a query it`s solved deadlock but in my case these subqueries are necessary.
Nested update with select deadlock
https://www.percona.com/blog/2017/09/25/avoid-shared-locks-from-subqueries-when-possible/
But these solutions don't solve my problem.
What I do wrong?
Maby someone had a similar problem and can help.
It is my UPDATE with subselect:
UPDATE mediaplan m
SET m.startDate = IFNULL((SELECT p.blockStartTime
FROM placement p
WHERE p.mediaplan_id = m.id AND p.deleted = FALSE
ORDER BY p.blockStartTime ASC
LIMIT 0, 1), NULL),
m.endDate = IFNULL((SELECT p.blockStartTime
FROM placement p
WHERE p.mediaplan_id = m.id AND p.deleted = FALSE
ORDER BY p.blockStartTime DESC
LIMIT 0, 1), NULL)
WHERE m.id IN (208396);
Deadlock log (SHOW ENGINE INNODB STATUS;)
=====================================
2018-03-28 15:00:52 0x700007086000 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 38 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 124 srv_active, 0 srv_shutdown, 9865 srv_idle
srv_master_thread log flush and writes: 9989
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 3037
OS WAIT ARRAY INFO: signal count 5042
RW-shared spins 0, rounds 7370, OS waits 1740
RW-excl spins 0, rounds 47629, OS waits 217
RW-sx spins 268, rounds 7782, OS waits 245
Spin rounds per wait: 7370.00 RW-shared, 47629.00 RW-excl, 29.04 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2018-03-28 15:00:32 0x700006f32000
*** (1) TRANSACTION:
TRANSACTION 1767639, ACTIVE 0 sec starting index read
mysql tables in use 3, locked 3
LOCK WAIT 6 lock struct(s), heap size 1136, 4 row lock(s), undo log entries 2
MySQL thread id 1105, OS thread handle 123145416400896, query id 17464 localhost 127.0.0.1 romexdev updating
UPDATE mediaplan m SET m.startDate = IFNULL( (SELECT p.blockStartTime FROM placement p WHERE p.mediaplan_id = m.id AND p.deleted = FALSE ORDER BY p.blockStartTime ASC LIMIT 0, 1), NULL ), m.endDate = IFNULL( (SELECT p.blockStartTime FROM placement p WHERE p.mediaplan_id = m.id AND p.deleted = FALSE ORDER BY p.blockStartTime DESC LIMIT 0, 1), NULL ) WHERE m.id IN (-1,208396)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1658 page no 541 n bits 320 index PRIMARY of table `openmarket`.`mediaplan` trx id 1767639 lock_mode X locks rec but not gap waiting
Record lock, heap no 16 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 4; hex 80032e0c; asc . ;;
1: len 6; hex 0000001af8d3; asc ;;
2: len 7; hex 7e000009480854; asc ~ H T;;
3: len 5; hex 99a006fc80; asc ;;
4: len 5; hex 999f748c00; asc t ;;
5: len 4; hex 8000027e; asc ~;;
6: SQL NULL;
7: len 1; hex 00; asc ;;
8: len 5; hex 999f6cc39f; asc l ;;
9: len 5; hex 999f6cc39f; asc l ;;
10: len 4; hex 800000a7; asc ;;
*** (2) TRANSACTION:
TRANSACTION 1767637, ACTIVE 0 sec fetching rows
mysql tables in use 3, locked 3
37 lock struct(s), heap size 3520, 900 row lock(s), undo log entries 2
MySQL thread id 1107, OS thread handle 123145418907648, query id 17455 localhost 127.0.0.1 romexdev Creating sort index
UPDATE mediaplan m SET m.startDate = IFNULL( (SELECT p.blockStartTime FROM placement p WHERE p.mediaplan_id = m.id AND p.deleted = FALSE ORDER BY p.blockStartTime ASC LIMIT 0, 1), NULL ), m.endDate = IFNULL( (SELECT p.blockStartTime FROM placement p WHERE p.mediaplan_id = m.id AND p.deleted = FALSE ORDER BY p.blockStartTime DESC LIMIT 0, 1), NULL ) WHERE m.id IN (-1,208396)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1658 page no 541 n bits 320 index PRIMARY of table `openmarket`.`mediaplan` trx id 1767637 lock_mode X locks rec but not gap
Record lock, heap no 16 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 4; hex 80032e0c; asc . ;;
1: len 6; hex 0000001af8d3; asc ;;
2: len 7; hex 7e000009480854; asc ~ H T;;
3: len 5; hex 99a006fc80; asc ;;
4: len 5; hex 999f748c00; asc t ;;
5: len 4; hex 8000027e; asc ~;;
6: SQL NULL;
7: len 1; hex 00; asc ;;
8: len 5; hex 999f6cc39f; asc l ;;
9: len 5; hex 999f6cc39f; asc l ;;
10: len 4; hex 800000a7; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1752 page no 435722 n bits 240 index PRIMARY of table `openmarket`.`placement` trx id 1767637 lock mode S locks rec but not gap waiting
Record lock, heap no 144 PHYSICAL RECORD: n_fields 31; compact format; info bits 0
0: len 4; hex 80c66779; asc gy;;
1: len 5; hex 999fa60000; asc ;;
2: len 6; hex 0000001af8d7; asc ;;
3: len 7; hex 210000c0160979; asc ! y;;
4: len 5; hex 999f78c6f6; asc x ;;
5: len 1; hex 00; asc ;;
6: len 1; hex 83; asc ;;
7: SQL NULL;
8: len 4; hex 88cb341e; asc 4 ;;
9: len 4; hex 80032e0c; asc . ;;
10: len 4; hex 80033ce3; asc < ;;
11: len 1; hex 01; asc ;;
12: len 5; hex 999fa62440; asc $#;;
13: SQL NULL;
14: SQL NULL;
15: SQL NULL;
16: SQL NULL;
17: SQL NULL;
18: SQL NULL;
19: len 1; hex 00; asc ;;
20: len 5; hex 999f78c6f6; asc x ;;
21: len 5; hex 999f78f020; asc x ;;
22: len 4; hex 80000001; asc ;;
23: SQL NULL;
24: len 1; hex 01; asc ;;
25: len 1; hex 01; asc ;;
26: len 1; hex 00; asc ;;
27: len 1; hex 00; asc ;;
28: len 4; hex 8002afaf; asc ;;
29: len 4; hex 80000012; asc ;;
30: SQL NULL;
*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------
Trx id counter 1767643
Purge done for trx's n:o < 1767643 undo n:o < 0 state: running but idle
History list length 31
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 281479519987928, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519987024, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519986120, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519985216, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519984312, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519989736, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519988832, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519983408, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (read thread)
I/O thread 4 state: waiting for i/o request (read thread)
I/O thread 5 state: waiting for i/o request (read thread)
I/O thread 6 state: waiting for i/o request (write thread)
I/O thread 7 state: waiting for i/o request (write thread)
I/O thread 8 state: waiting for i/o request (write thread)
I/O thread 9 state: waiting for i/o request (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
508449 OS file reads, 49980 OS file writes, 15796 OS fsyncs
0.24 reads/s, 16384 avg bytes/read, 2.58 writes/s, 0.76 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 3092, seg size 3094, 84 merges
merged operations:
insert 134, delete mark 3, delete 1
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 34673, node heap has 4 buffer(s)
Hash table size 34673, node heap has 24 buffer(s)
Hash table size 34673, node heap has 62 buffer(s)
Hash table size 34673, node heap has 201 buffer(s)
Hash table size 34673, node heap has 73 buffer(s)
Hash table size 34673, node heap has 2 buffer(s)
Hash table size 34673, node heap has 1 buffer(s)
Hash table size 34673, node heap has 2 buffer(s)
133.47 hash searches/s, 26.13 non-hash searches/s
---
LOG
---
Log sequence number 212087243581
Log flushed up to 212087243581
Pages flushed up to 212087243581
Last checkpoint at 212087243572
0 pending log flushes, 0 pending chkp writes
678 log i/o's done, 0.39 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 137428992
Dictionary memory allocated 5812878
Buffer pool size 8191
Free buffers 1024
Database pages 6798
Old database pages 2489
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 24521, not young 13076464
0.00 youngs/s, 1.39 non-youngs/s
Pages read 504777, created 29725, written 31405
0.24 reads/s, 0.00 creates/s, 2.11 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 3 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 6798, unzip_LRU len: 0
I/O sum[130]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=36398, Main thread ID=123145409064960, state: sleeping
Number of rows inserted 49479, updated 123, deleted 0, read 16253106
0.08 inserts/s, 0.16 updates/s, 0.00 deletes/s, 276.97 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
Try
BEGIN;
SELECT #start := ... LIMIT 1;
SELECT #end := ... LIMIT 1;
UPDATE ...
SET startDate = COALESCE(startDate, #start),
endDate = COALESCE(startDate, #end)
WHERE ...
COMMIT;
I changed my select statement transaction to read uncommitted but still it holds shared lock and get involved in deadlock with another update statement,
Select statements are in whatsnew(uid,friend_news_id,actor_news_id) SP
SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED ;
set #actor_news_limit = 2;
set #player_news_limit = 6;
if (select
n.id
from
news n
inner join
news_user_xrf nu
on
n.id = nu.news_id
inner join
friends f
on
nu.user_id = f.friend_id
inner join
user u
on
u.id = f.friend_id
where
f.user_id = uid
and
n.id > friend_news_id
order by
n.points desc,n.newsDate desc
limit 0,1) is null
then
select * from
(select
n.id,
'whatshot_friendsnews' as type,
n.content,
'0' as actor_id,
u.id as player_id,
n.newsType,
n.like_text,
n.link,
n.newsDate,
u.imgurl,
u.firstname,
u.lastname,
n.points
from
news n
inner join
news_user_xrf nu
on
n.id = nu.news_id
inner join
user u
on
u.id = nu.user_id
where
u.id <> uid
and
n.id > friend_news_id
order by
n.newsDate desc
limit 0,3) a order by newsDate;
else
select * from
(select
n.id,
'whatshot_friendsnews' as type,
n.content,
'0' as actor_id,
u.id as player_id,
n.newsType,
n.like_text,
n.link,
n.newsDate,
u.imgurl,
u.firstname,
u.lastname,
n.points
from
news n
inner join
news_user_xrf nu
on
n.id = nu.news_id
inner join
friends f
on
nu.user_id = f.friend_id
inner join
user u
on
u.id = f.friend_id
where
f.user_id = uid
and
n.id > friend_news_id
order by
n.points desc,n.newsDate desc
limit 0,3) a order by newsDate;
end if;
select * from
(select
n.id,
'whatshot_actornews' as type,
n.content,
na.actor_id,
"0" as player_id,
n.newsType,
n.like_text,
n.link,
n.newsDate,
n.newsImg as imgurl,
'' as firstname,
'' as lastname,
n.points
from
news n
inner join
news_actor_xrf na
on
n.id = na.news_id
where
newsType = 2
and
n.id > actor_news_id
order by
points desc,
newsDate desc
limit
0,3
) a order by newsDate;
COMMIT ;
END
now i have another update statement where i update same table and this statement wait for X lock but it cannot have it as the whatsnew sp holds S lock on the same table,
below is Update sp
set SQL_SAFE_UPDATES=0;
update user
set networth = (select sum(p.avgcostprice * p.quantity)
from portfolio p where p.user_id = user.id) +
(select truncate(sum((a.currentvalue - p.avgcostprice)*p.quantity),2)
from actor a inner join portfolio p on a.id = p.actor_id where p.user_id = user.id) +
user.balance;
update user u
set u.rank = (select count(*) from friends f
where ifnull(friend_networth,0) > user_networth
and f.user_id = u.id) + 1;
update user u
set u.rank_text = (select concat(u.rank,'/',(select
count(*)+1 from friends f where f.user_id = u.id)));
Now both of these stored procedure gets involved in deadlock, but question is even when i am using read uncommitted why select statement
holds Shared lock on the rows, it should read( dirty read) from the
table. here is SHOW ENGINE INNODB STATUS output
=====================================
2014-07-06 03:44:57 2ad677681700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 2 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 32811 srv_active, 0 srv_shutdown, 89228 srv_idle
srv_master_thread log flush and writes: 122039
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 22814
OS WAIT ARRAY INFO: signal count 21778
Mutex spin waits 6627, rounds 508985, OS waits 15885
RW-shared spins 5876, rounds 176305, OS waits 5659
RW-excl spins 44, rounds 38079, OS waits 1237
Spin rounds per wait: 76.80 mutex, 30.00 RW-shared, 865.43 RW-excl
------------------------
LATEST DETECTED DEADLOCK
------------------------
2014-07-06 02:01:41 2ad677681700
*** (1) TRANSACTION:
TRANSACTION 1925770, ACTIVE 0 sec fetching rows
mysql tables in use 2, locked 2
LOCK WAIT 25 lock struct(s), heap size 3112, 909 row lock(s)
MySQL thread id 28709, OS thread handle 0x2ad677640700, query id 349542 ec2-54-226-145-7.compute-1.amazonaws.com 54.227.116.3 Database1 Searching rows for update
update user u
set u.rank_text = (select concat(u.rank,''/'',(select
count(*)+1 from friends f where f.user_id = u.id)))
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 19303 page no 29 n bits 104 index `PRIMARY` of table `Database1`.`user` trx id 1925770 lock_mode X waiting
Record lock, heap no 31 PHYSICAL RECORD: n_fields 25; compact format; info bits 0
0: len 8; hex 8000000000000416; asc ;;
1: len 6; hex 0000001d214a; asc !J;;
2: len 7; hex 6f0000018d08f9; asc o ;;
3: len 8; hex 80005af4001478e8; asc Z x ;;
4: len 5; hex 5269736869; asc Rishi;;
5: len 5; hex 506174656c; asc Patel;;
6: len 23; hex 7269736869706174656c5f3939407961686f6f2e636f6d; asc rishi_9999#yahoo.com;;
7: len 30; hex 43414146336348376866544942414a434844466b5a42664937544a554643; asc CAAF3cH7hfTIBAJCHDFkZBfI7TJUFC; (total 201 bytes);
8: len 8; hex 800000005405f359; asc T Y;;
9: len 9; hex 80000000000007d619; asc ;;
10: len 1; hex 01; asc ;;
11: len 30; hex 313034383539303739377c3130303030303937303735313938317c313030; asc 1048590797|100000970751981|100; (total 58 bytes);
12: len 30; hex 68747470733a2f2f67726170682e66616365626f6f6b2e636f6d2f313030; asc https://graph.facebook.com/100; (total 50 bytes);
13: SQL NULL;
14: len 4; hex 80000002; asc ;;
15: len 4; hex 466f4145; asc FoAE;;
16: SQL NULL;
17: len 4; hex 00000000; asc ;;
18: len 3; hex 322f35; asc 2/5;;
19: SQL NULL;
20: len 8; hex 8000000000000007; asc ;;
21: len 4; hex 05c58045; asc E;;
22: SQL NULL;
23: len 8; hex 8000000000000000; asc ;;
24: len 8; hex 8000000000000007; asc ;;
*** (2) TRANSACTION:
TRANSACTION 1925761, ACTIVE 22 sec starting index read
mysql tables in use 4, locked 4
14 lock struct(s), heap size 3112, 4 row lock(s)
MySQL thread id 28723, OS thread handle 0x2ad677681700, query id 349517 ec2-54-91-34-208.compute-1.amazonaws.com 54.91.34.208 Database1 Sending data
call whatsnew(uid,friend_news_id,actor_news_id)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 19303 page no 29 n bits 104 index `PRIMARY` of table `Database1`.`user` trx id 1925761 lock mode S locks rec but not gap
Record lock, heap no 31 PHYSICAL RECORD: n_fields 25; compact format; info bits 0
0: len 8; hex 8000000000000416; asc ;;
1: len 6; hex 0000001d214a; asc !J;;
2: len 7; hex 6f0000018d08f9; asc o ;;
3: len 8; hex 80005af4001478e8; asc Z x ;;
4: len 5; hex 5269736869; asc Rishi;;
5: len 5; hex 506174656c; asc Patel;;
6: len 23; hex 7269736869706174656c5f3939407961686f6f2e636f6d; asc rishi_9999#yahoo.com;;
7: len 30; hex 43414146336348376866544942414a434844466b5a42664937544a554643; asc CAAF3cH7hfTIBAJCHDFkZBfI7TJUFC; (total 201 bytes);
8: len 8; hex 800000005405f359; asc T Y;;
9: len 9; hex 80000000000007d619; asc ;;
10: len 1; hex 01; asc ;;
11: len 30; hex 313034383539303739377c3130303030303937303735313938317c313030; asc 1048590797|100000970751981|100; (total 58 bytes);
12: len 30; hex 68747470733a2f2f67726170682e66616365626f6f6b2e636f6d2f313030; asc https://graph.facebook.com/100; (total 50 bytes);
13: SQL NULL;
14: len 4; hex 80000002; asc ;;
15: len 4; hex 466f4145; asc FoAE;;
16: SQL NULL;
17: len 4; hex 00000000; asc ;;
18: len 3; hex 322f35; asc 2/5;;
19: SQL NULL;
20: len 8; hex 8000000000000007; asc ;;
21: len 4; hex 05c58045; asc E;;
22: SQL NULL;
23: len 8; hex 8000000000000000; asc ;;
24: len 8; hex 8000000000000007; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 19303 page no 29 n bits 104 index `PRIMARY` of table `Database1`.`user` trx id 1925761 lock mode S locks rec but not gap waiting
Record lock, heap no 26 PHYSICAL RECORD: n_fields 25; compact format; info bits 0
0: len 8; hex 800000000000040b; asc ;;
1: len 6; hex 0000001d214a; asc !J;;
2: len 7; hex 6f0000018d0878; asc o x;;
3: len 8; hex 80005af401790e35; asc Z y 5;;
4: len 3; hex 52616a; asc Raj;;
5: len 5; hex 506174656c; asc Patel;;
6: len 22; hex 706174656c3039303972616a407961686f6f2e636f6d; asc patel#yahoo.com;;
7: len 30; hex 434141463363483768665449424145726535315545495475473766556652; asc CAAF3cH7hfTIBAEre51UEITuG7fUfR; (total 199 bytes);
8: len 8; hex 8000000054060653; asc T S;;
9: len 9; hex 800000000000098440; asc #;;
10: len 1; hex 01; asc ;;
11: len 30; hex 313034383539303739377c313432343530303835307c3130303030303937; asc 1048590797|1424500850|10000097; (total 85 bytes);
12: len 30; hex 68747470733a2f2f67726170682e66616365626f6f6b2e636f6d2f313030; asc graph.facebook.com/10000; (total 50 bytes);
13: SQL NULL;
14: len 4; hex 80000001; asc ;;
15: SQL NULL;
16: SQL NULL;
17: len 4; hex 00000000; asc ;;
18: len 3; hex 312f37; asc 1/7;;
19: len 4; hex 00000000; asc ;;
20: len 8; hex 8000000000000018; asc ;;
21: len 4; hex 6bdb7e45; asc k ~E;;
22: SQL NULL;
23: len 8; hex 8000000000000000; asc ;;
24: len 8; hex 8000000000000018; asc ;;
*** WE ROLL BACK TRANSACTION (2)
------------
TRANSACTIONS
------------
Trx id counter 1929046
Purge done for trx''s n:o < 1929045 undo n:o < 0 state: running but idle
History list length 1711
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 28864, OS thread handle 0x2ad677681700, query id 357530 122.169.237.65 Database1 init
SHOW ENGINE INNODB STATUS
---TRANSACTION 1929043, not started
MySQL thread id 28863, OS thread handle 0x2ad677640700, query id 357529 ec2-54-226-145-7.compute-1.amazonaws.com 54.227.116.3 Database1 creating table
create table temp_ceiling (actor_id int, p_change float, cval float, fval float)
---TRANSACTION 1929042, not started
MySQL thread id 28859, OS thread handle 0x2ad675ee3700, query id 357510 localhost 127.0.0.1 rdsadmin cleaning up
---TRANSACTION 1929045, ACTIVE 0 sec committing foreign key definitions
5 lock struct(s), heap size 1248, 0 row lock(s), undo log entries 8
MySQL thread id 28863, OS thread handle 0x2ad677640700, query id 357529 ec2-54-226-145-7.compute-1.amazonaws.com 54.227.116.3 Database1 creating table
create table temp_ceiling (actor_id int, p_change float, cval float, fval float)
TABLE LOCK table `SYS_TABLES` trx id 1929045 lock mode IX
TABLE LOCK table `SYS_COLUMNS` trx id 1929045 lock mode IX
TABLE LOCK table `SYS_TABLESPACES` trx id 1929045 lock mode IX
TABLE LOCK table `SYS_DATAFILES` trx id 1929045 lock mode IX
TABLE LOCK table `SYS_INDEXES` trx id 1929045 lock mode IX
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o''s: 0, sync i/o''s: 0
Pending flushes (fsync) log: 0; buffer pool: 0
946 OS file reads, 122080 OS file writes, 61052 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 4000.00 writes/s, 5000.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 612301, node heap has 33 buffer(s)
0.00 hash searches/s, 5.00 non-hash searches/s
---
LOG
---
Log sequence number 502059361
Log flushed up to 502055320
Pages flushed up to 502053286
Last checkpoint at 502053286
0 pending log writes, 0 pending chkp writes
39359 log i/o''s done, 1.00 log i/o''s/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 316588032; in additional pool allocated 0
Dictionary memory allocated 269802
Buffer pool size 18880
Free buffers 11148
Database pages 7699
Old database pages 2822
Modified db pages 16
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 886, created 6813, written 73085
0.00 reads/s, 4000.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 7699, unzip_LRU len: 0
I/O sum[0]:cur[1], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Main thread process no. 18631, id 47100585178880, state: sleeping
Number of rows inserted 43271, updated 151830, deleted 2009, read 7297650700
0.00 inserts/s, 14.49 updates/s, 0.00 deletes/s, 167000.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
'
please help
I have a Mediacore-based website that is getting stuck due to an update query. Those update queries are running all the time, but sometimes (I believe, during high traffic hours), one of those updates just becomes stuck, locking all other queries:
245 mediacore_user localhost mediacore Query 36 Locked UPDATE media SET modified_on='2011-12-21 09:42:58', views=(media.views + 1) WHERE media.id = 1048
246 mediacore_user localhost mediacore Query 36 Locked UPDATE media SET modified_on='2011-12-21 09:42:58', views=(media.views + 1) WHERE media.id = 1048
248 mediacore_user localhost mediacore Query 36 Updating UPDATE media SET modified_on='2011-12-21 09:42:58', views=(media.views + 1) WHERE media.id = 1048
249 mediacore_user localhost mediacore Query 36 Locked UPDATE media SET modified_on='2011-12-21 09:42:58', views=(media.views + 1) WHERE media.id = 1048
253 mediacore_user localhost mediacore Query 36 Locked UPDATE media SET modified_on='2011-12-21 09:42:58', views=(media.views + 1) WHERE media.id = 1048
258 mediacore_user localhost mediacore Query 30 Locked UPDATE media SET modified_on='2011-12-21 09:43:04', views=(media.views + 1) WHERE media.id = 416
268 mediacore_user localhost mediacore Query 3 Locked UPDATE media SET modified_on='2011-12-21 09:43:31', views=(media.views + 1) WHERE media.id = 1048
269 mediacore_user localhost mediacore Query 3 Locked UPDATE media SET modified_on='2011-12-21 09:43:31', views=(media.views + 1) WHERE media.id = 1048
270 mediacore_user localhost mediacore Query 3 Locked UPDATE media SET modified_on='2011-12-21 09:43:31', views=(media.views + 1) WHERE media.id = 1048
271 mediacore_user localhost mediacore Query 3 Locked UPDATE media SET modified_on='2011-12-21 09:43:31', views=(media.views + 1) WHERE media.id = 1048
272 mediacore_user localhost mediacore Query 1 Locked UPDATE media SET modified_on='2011-12-21 09:43:33', views=(media.views + 1) WHERE media.id = 1048
If I kill the "Updating" connection, another one gets stuck, and so on, until there are only about 3 queries left locked.
Any ideas?
Media table:
CREATE TABLE `media` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`type` varchar(8) DEFAULT NULL,
`slug` varchar(50) NOT NULL,
`podcast_id` int(11) DEFAULT NULL,
`reviewed` tinyint(1) NOT NULL,
`encoded` tinyint(1) NOT NULL,
`publishable` tinyint(1) NOT NULL,
`created_on` datetime NOT NULL,
`modified_on` datetime NOT NULL,
`publish_on` datetime DEFAULT NULL,
`publish_until` datetime DEFAULT NULL,
`title` varchar(255) NOT NULL,
`subtitle` varchar(255) DEFAULT NULL,
`description` text,
`description_plain` text,
`notes` text,
`duration` int(11) NOT NULL,
`views` int(11) NOT NULL,
`likes` int(11) NOT NULL,
`popularity_points` int(11) NOT NULL,
`author_name` varchar(50) NOT NULL,
`author_email` varchar(255) NOT NULL,
`dislikes` int(11) NOT NULL,
`popularity_likes` int(11) NOT NULL,
`popularity_dislikes` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `slug` (`slug`),
KEY `podcast_id` (`podcast_id`),
CONSTRAINT `media_ibfk_1` FOREIGN KEY (`podcast_id`) REFERENCES `podcasts` (`id`) ON DELETE SET NULL ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=1049 DEFAULT CHARSET=utf8
INNODB Status after the problem happened:
=====================================
111221 12:12:09 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 6 seconds
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 78, signal count 78
Mutex spin waits 0, rounds 453, OS waits 17
RW-shared spins 122, OS waits 61; RW-excl spins 0, OS waits 0
------------
TRANSACTIONS
------------
Trx id counter 0 1115749
Purge done for trx's n:o < 0 1115735 undo n:o < 0 0
History list length 13
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0 1115747, not started, process no 15360, OS thread id 140486827099904
MySQL thread id 111, query id 11647 localhost mediacore_user
---TRANSACTION 0 0, not started, process no 15360, OS thread id 140486827898624
MySQL thread id 106, query id 11651 localhost root
show innodb status
---TRANSACTION 0 1115709, not started, process no 15360, OS thread id 140486826567424
MySQL thread id 103, query id 10874 localhost mediacore_user
---TRANSACTION 0 1115731, not started, process no 15360, OS thread id 140486792349440
MySQL thread id 102, query id 11517 localhost mediacore_user
---TRANSACTION 0 1115668, not started, process no 15360, OS thread id 140486792881920
MySQL thread id 101, query id 10067 localhost 127.0.0.1 a7user
---TRANSACTION 0 1115737, not started, process no 15360, OS thread id 140486791816960
MySQL thread id 95, query id 11578 localhost mediacore_user
---TRANSACTION 0 1115738, not started, process no 15360, OS thread id 140486827366144
MySQL thread id 87, query id 11577 localhost mediacore_user
---TRANSACTION 0 1115744, not started, process no 15360, OS thread id 140486827632384
MySQL thread id 61, query id 11606 localhost mediacore_user
---TRANSACTION 0 1115746, not started, process no 15360, OS thread id 140486826301184
MySQL thread id 22, query id 11633 localhost mediacore_user
---TRANSACTION 0 1115748, ACTIVE 1 sec, process no 15360, OS thread id 140486826833664
mysql tables in use 1, locked 1
MySQL thread id 55, query id 11650 localhost mediacore_user Table lock
UPDATE media SET modified_on='2011-12-21 12:12:08', dislikes=10 WHERE media.id = 572
Trx read view will not see trx with id >= 0 1115749, sees < 0 1115740
---TRANSACTION 0 1115743, ACTIVE 14 sec, process no 15360, OS thread id 140486826034944
mysql tables in use 1, locked 1
MySQL thread id 23, query id 11601 localhost mediacore_user Table lock
UPDATE media SET modified_on='2011-12-21 12:11:55', views=(media.views + 1) WHERE media.id = 1048
Trx read view will not see trx with id >= 0 1115744, sees < 0 1115740
---TRANSACTION 0 1115742, ACTIVE 14 sec, process no 15360, OS thread id 140486825502464
mysql tables in use 1, locked 1
MySQL thread id 90, query id 11599 localhost mediacore_user Table lock
UPDATE media SET modified_on='2011-12-21 12:11:55', views=(media.views + 1) WHERE media.id = 1048
Trx read view will not see trx with id >= 0 1115743, sees < 0 1115740
---TRANSACTION 0 1115741, ACTIVE 14 sec, process no 15360, OS thread id 140486825768704 starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 368, 1 row lock(s)
MySQL thread id 73, query id 11596 localhost mediacore_user Updating
UPDATE media SET modified_on='2011-12-21 12:11:55', views=(media.views + 1) WHERE media.id = 1048
Trx read view will not see trx with id >= 0 1115742, sees < 0 1115740
------- TRX HAS BEEN WAITING 14 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 1025 n bits 88 index `PRIMARY` of table `mediacore`.`media` trx id 0 1115741 lock_mode X locks rec but not gap waiting
Record lock, heap no 20 PHYSICAL RECORD: n_fields 27; compact format; info bits 0
0: len 4; hex 80000418; asc ;; 1: len 6; hex 00000011065c; asc \;; 2: len 7; hex 00000003c20d5b; asc [;; 3: len 5; hex 766964656f; asc video;; 4: len 30; hex 617273656c696f2d7061746f2d64652d63617276616c686f2d686f6d656e; asc arselio-pato-de-carvalho-homen;...(truncated); 5: len 4; hex 80000013; asc ;; 6: len 1; hex 81; asc ;; 7: len 1; hex 81; asc ;; 8: len 1; hex 81; asc ;; 9: len 8; hex 8000124a82217b0b; asc J !{ ;; 10: len 8; hex 8000124a8230e083; asc J 0 ;; 11: len 8; hex 8000124a82220ef0; asc J " ;; 12: SQL NULL; 13: len 30; hex 417273c3a96c696f205061746f2064652043617276616c686f20686f6d65; asc Ars lio Pato de Carvalho home;...(truncated); 14: SQL NULL; 15: len 30; hex 3c703e43656e74726f206465204e6575726f6369c3aa6e63696173206520; asc <p>Centro de Neuroci ncias e ;...(truncated); 16: len 30; hex 43656e74726f206465204e6575726f6369c3aa6e6369617320652042696f; asc Centro de Neuroci ncias e Bio;...(truncated); 17: SQL NULL; 18: len 4; hex 80000000; asc ;; 19: len 4; hex 80000099; asc ;; 20: len 4; hex 80000000; asc ;; 21: len 4; hex 80000000; asc ;; 22: len 3; hex 554356; asc UCV;; 23: len 9; hex 7563764075632e7074; asc ucv#uc.pt;; 24: len 4; hex 80000000; asc ;; 25: len 4; hex 80000000; asc ;; 26: len 4; hex 80000000; asc ;;
------------------
---TRANSACTION 0 1115740, ACTIVE 14 sec, process no 15360, OS thread id 140486792083200
2 lock struct(s), heap size 368, 1 row lock(s), undo log entries 1
MySQL thread id 97, query id 11597 localhost mediacore_user Table lock
SELECT 1
FROM media_fulltext
LIMIT 1
Trx read view will not see trx with id >= 0 1115741, sees < 0 1115741
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (write thread)
Pending normal aio reads: 0, aio writes: 0,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
187 OS file reads, 469 OS file writes, 298 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2,
0 inserts, 0 merged recs, 0 merges
Hash table size 2212699, node heap has 9 buffer(s)
0.17 hash searches/s, 0.17 non-hash searches/s
---
LOG
---
Log sequence number 0 148655397
Log flushed up to 0 148655397
Last checkpoint at 0 148655397
0 pending log writes, 0 pending chkp writes
177 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 1225697720; in additional pool allocated 11846912
Dictionary memory allocated 203840
Buffer pool size 65536
Free buffers 65299
Database pages 228
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages read 225, created 3, written 259
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
6 read views open inside InnoDB
Main thread process no. 15360, id 140486801540864, state: sleeping
Number of rows inserted 13, updated 85, deleted 2, read 281971
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.17 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
EDIT: Somehow, some of the updates are not unlocking the table after they commit. I watched the queries for quite some time and saw that after 50 seconds, the lock is freed and one update is executed. I've changed the innodb_lock_wait_timeout to 5 and I saw the same thing, only with intervals of 5 seconds instead of 50. But once it gets only a few updates left locked, they're all executed instantaneously... it's a very strange behavior...
A bit late but I'd like to add some more information: From my observation this happens if you have the full-text index enabled. In that case there seems to be a bad interaction in MySQL involving InnoDB (media table) and MyISAM (full-text index) locking.
I never saw the issue if MediaCore's full-text triggers were not installed.
I think it is because of the large number of queries at a time. Try using mysql index with your fields. It will speed up the performance.