MYSQL slow performance because of Locking - mysql

I have Mysql5.5 on windows system. Since past 1 month it is running slow.
Here is the query which are running the most:
begin
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1
where roomlogs.netId = pnetid
and roomlogs.`status` = 0
and roomlogs.enterTime>=DATE(ptime)
and roomlogs.enterTime<DATE(ptime) + interval 1 day;
INSERT
into roomlogs (roomlogs.roomId, roomlogs.netId, roomlogs.enterTime,
roomlogs.exitTime, roomlogs.hidden, roomlogs.checkinId)
value
(pRoomId, pnetid,ptime, pexit, phidden, pcheckid);
select *
from roomlogs
where roomlogs.roomlogId = LAST_INSERT_ID();
end
AND:
begin
select count(*) as total
from roomlogs
where roomId = proomId
and roomlogs.`status` = 0
AND roomlogs.enterTime >= CURDATE()
AND roomlogs.enterTime < CURDATE() + INTERVAL 1 DAY
and roomlogs.hidden!=1;
end
Here index of roomlogs are primary(roomlogId), NETID(netid,status,enterTime), ROOMID(roomID,status,enterTime)
Here is my InnoDB status when MySQL has more number of queries (around 100) running simultaneously in the server.
=====================================
2018-10-25 17:42:01 1728 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 20 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 148860 srv_active, 0 srv_shutdown, 271514 srv_idle
srv_master_thread log flush and writes: 420374
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 80056
OS WAIT ARRAY INFO: signal count 80039
Mutex spin waits 6920930, rounds 7674964, OS waits 52940
RW-shared spins 27042, rounds 803211, OS waits 26514
RW-excl spins 1173, rounds 9109, OS waits 211
Spin rounds per wait: 1.11 mutex, 29.70 RW-shared, 7.77 RW-excl
------------
TRANSACTIONS
------------
Trx id counter 78706084
Purge done for trx's n:o < 78706079 undo n:o < 0 state: running but idle
History list length 2281
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 78705887, not started
MySQL thread id 3000, OS thread handle 0x8e0, query id 4915274 localhost 127.0.0.1 root cleaning up
**------------ SKIPPING SOME DATA HERE--------------**
---TRANSACTION 78701492, not started
MySQL thread id 15, OS thread handle 0x12c4, query id 4889345 localhost 127.0.0.1 root cleaning up
---TRANSACTION 78697207, not started
MySQL thread id 16, OS thread handle 0x408, query id 4889123 localhost 127.0.0.1 root cleaning up
---TRANSACTION 78701491, not started
MySQL thread id 14, OS thread handle 0x11c8, query id 4889320 localhost 127.0.0.1 root cleaning up
---TRANSACTION 78697202, not started
MySQL thread id 13, OS thread handle 0x10dc, query id 4889098 localhost 127.0.0.1 root cleaning up
---TRANSACTION 78706073, ACTIVE 1 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2989, OS thread handle 0x157c, query id 4916082 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 1 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78706073 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78706056, ACTIVE 4 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2942, OS thread handle 0x15e0, query id 4916017 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 4 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78706056 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78706045, ACTIVE 5 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2976, OS thread handle 0x125c, query id 4915976 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 5 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78706045 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78706009, ACTIVE 9 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2946, OS thread handle 0xb98, query id 4915824 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 9 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78706009 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78706004, ACTIVE 10 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2995, OS thread handle 0x1234, query id 4915806 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 10 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78706004 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78705997, ACTIVE 12 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2948, OS thread handle 0x1708, query id 4915783 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 12 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78705997 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78705985, ACTIVE 14 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 2986, OS thread handle 0x1640, query id 4915747 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
------- TRX HAS BEEN WAITING 14 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1461 page no 6 n bits 328 index `PRIMARY` of table `indoormap_v2_beta4`.`roomlogs` trx id 78705985 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 80000011; asc ;;
1: len 6; hex 0000028a4fc7; asc O ;;
2: len 7; hex c3000001ed0110; asc ;;
3: len 7; hex 726f6f6d313038; asc room108;;
4: len 7; hex 7973756e313032; asc ysun102;;
5: len 5; hex 999d1f25db; asc % ;;
6: len 5; hex 999d1d2a13; asc * ;;
7: len 4; hex 80000001; asc ;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 7fffffff; asc ;;
------------------
---TRANSACTION 78705977, ACTIVE 16 sec fetching rows
mysql tables in use 1, locked 1
4610 lock struct(s), heap size 539064, 1001945 row lock(s)
MySQL thread id 2938, OS thread handle 0x10bc, query id 4915725 localhost 127.0.0.1 root updating
update roomlogs set roomlogs.exitTime = ptime,roomlogs.`status` = 1 where roomlogs.netId = pnetid and roomlogs.`status` = 0 and
roomlogs.enterTime>=DATE(ptime) and
roomlogs.enterTime<DATE(ptime) + interval 1 day
--------
FILE I/O
--------
I/O thread 0 state: wait Windows aio (insert buffer thread)
I/O thread 1 state: wait Windows aio (log thread)
I/O thread 2 state: wait Windows aio (read thread)
I/O thread 3 state: wait Windows aio (read thread)
I/O thread 4 state: wait Windows aio (read thread)
I/O thread 5 state: wait Windows aio (read thread)
I/O thread 6 state: wait Windows aio (write thread)
I/O thread 7 state: wait Windows aio (write thread)
I/O thread 8 state: wait Windows aio (write thread)
I/O thread 9 state: wait Windows aio (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
23467 OS file reads, 931766 OS file writes, 524775 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 1.00 writes/s, 1.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 39, seg size 41, 61 merges
merged operations:
insert 57, delete mark 5, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 4425293, node heap has 2003 buffer(s)
30.00 hash searches/s, 19.95 non-hash searches/s
---
LOG
---
Log sequence number 7396884835
Log flushed up to 7396884835
Pages flushed up to 7396717673
Last checkpoint at 7396717673
0 pending log writes, 0 pending chkp writes
164342 log i/o's done, 1.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 2197815296; in additional pool allocated 0
Dictionary memory allocated 2793588
Buffer pool size 131072
Free buffers 101935
Database pages 27102
Old database pages 9967
Modified db pages 377
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 498, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 22500, created 4602, written 613885
0.00 reads/s, 0.10 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 27102, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 16384
Free buffers 12818
Database pages 3312
Old database pages 1240
Modified db pages 63
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 72, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2713, created 599, written 81358
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3312, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 16384
Free buffers 12809
Database pages 3319
Old database pages 1215
Modified db pages 27
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 15, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2770, created 549, written 78718
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3319, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 16384
Free buffers 12717
Database pages 3411
Old database pages 1256
Modified db pages 28
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 77, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2795, created 616, written 49290
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3411, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size 16384
Free buffers 12733
Database pages 3395
Old database pages 1246
Modified db pages 51
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 45, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2780, created 615, written 91680
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3395, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size 16384
Free buffers 12812
Database pages 3318
Old database pages 1217
Modified db pages 47
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 108, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2645, created 673, written 95064
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3318, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size 16384
Free buffers 12590
Database pages 3538
Old database pages 1294
Modified db pages 51
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 35, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2905, created 633, written 89554
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3538, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size 16384
Free buffers 12563
Database pages 3567
Old database pages 1307
Modified db pages 52
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 32, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 3127, created 440, written 56311
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3567, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size 16384
Free buffers 12893
Database pages 3242
Old database pages 1192
Modified db pages 58
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 114, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2765, created 477, written 71910
0.00 reads/s, 0.10 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3242, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
1 read views open inside InnoDB
Main thread id 5080, state: sleeping
Number of rows inserted 97780, updated 862074, deleted 219, read 468159566176
1.55 inserts/s, 1.20 updates/s, 0.00 deletes/s, 3363648.47 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
So, Here you can see that transaction was waiting for a record lock, Also the last transaction with ID 78705977 is locking 1001945 rows which it should be as it is not updating that much rows.
Also Why it is taking time to fetch the rows like 16 seconds?
Here is create Table of roomlogs:
CREATE TABLE `roomlogs` (
`roomId` varchar(200) CHARACTER SET latin1 DEFAULT NULL,
`netId` varchar(50) CHARACTER SET latin1 DEFAULT NULL,
`enterTime` datetime DEFAULT NULL,
`exitTime` datetime DEFAULT NULL,
`roomlogId` int(11) NOT NULL AUTO_INCREMENT,
`status` int(11) DEFAULT '0',
`hidden` int(11) DEFAULT '0',
`checkinId` int(11) DEFAULT '-1',
PRIMARY KEY (`roomlogId`),
KEY `NETID` (`netId`,`status`,`enterTime`),
KEY `RoomLogIndex` (`roomId`,`status`,`enterTime`)
) ENGINE=InnoDB AUTO_INCREMENT=1315726 DEFAULT CHARSET=utf8

Normally the contention should not happen because of the status - attribute which is updated at the same time.
But because of MVCC the different transaction do not recognize that the records are not to be changed anymore. Therefore they all see the same records and lock them even if they would not have to do that because of the already changed status.
I suspect, that for you it is important, that the single change is done.
At the moment the transaction A (for example) locks all the records as they seem to be changed, according to the state of the data beginning of the transaction.
As soon as all records could be locked by A because all transactions running parallel at the beginning have been committed, transaction A locks and fetches those records, finds out, that the status has already been changed and does the changes on the leftovers.
After that change the originally locked set of records are unlocked.
The leftovers might not be many, but the contention is heavy.
The solution I see:
There seem to be many parallel threads changing the records with the same netid. Can that be avoided?
if not, I never tried that, but perhaps changing the isolation level can help: See: https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html

I don't (yet) see why the update is stumbling over itself. But I see some things that might speed up other things, thereby helping indirectly:
Item 1, faster select
select *
from roomlogs
where roomlogs.roomlogId = LAST_INSERT_ID();
Could (perhaps) be replaced by
SELECT LAST_INSERT_ID(),
pRoomId, pnetid, ptime, pexit, phidden, pcheckid;
Caution: I am assuming that those are the only columns, and that the order is correct. Please provide SHOW CREATE TABLE roomlogs so we don't have to guess.
Item 2, covering index
INDEX (roomID, status, enterTime)
-->
INDEX (roomID, status, enterTime, hidden)
This would be a "covering index", thereby speeding up the SELECT COUNT(*) ...
Item 3, better optimization
If hidden has only values 0 and 1, then
hidden != 1
-->
hidden = 0
The Optimizer can handle = better than !=. At that point, this is an even better index for item 2:
INDEX (roomID, status, hidden, enterTime)
Note: I did not recommend this in Item 2 because of the !=.
Item 4, datatype:
If status is not some form of INT, then I will have the answer. Again, need SHOW CREATE TABLE roomlogs

Something to try:
begin
START TRANSACTION;
SELECT *
FROM roomlogs
WHERE netId = pnetid
AND `status` = 0
AND enterTime >= DATE(ptime)
AND enterTime < DATE(ptime) + interval 1 day;
FOR UPDATE; -- This might cause different locking action
-- WHERE clause is equivalent to UPDATE's
-- The following are as before (or as further optimized):
UPDATE ...
INSERT ...
SELECT ...
COMMIT;
end;
A possible reason: In older versions of MySQL, a different, less sophisticated, Optimizer was used in UPDATE and DELETE. Hence, upgrading might be a solution.
Check for errors. You could get a timeout or deadlock. Either need to be handled.

(Yeah, I know this is my 3rd answer. But it is in a different direction.)
Consider moving status out of the table. This is especially since multiple rows need to be changed. Also since you have a datetime, but the status is associated with a date.
The new table would have
roomid ...
status TINYINT UNSIGNED NOT NULL (default 0?)
date DATE NOT NULL -- comes from DATE(enterTime)
PRIMARY KEY(roomid)
You would JOIN to the table when you need to see the status.
You would update the one row when you need to change the status. And you would not necessarily touch the other table.
This idea will probably fail if you need some rows to be status=0 and some status=1 simultaneously. However, if only 'older' rows have =0, then adding a time to the table would say "status is 0 before the time". This would make the JOIN a bit more complex, but still doable.
Please tell us what status "means". And consider changing the column name to reflect the semantics.

Related

Spring Boot Hibernate's optimistic lock case deadlock in MySQL

I have a question about the deadlock from optimistic lock.
I implemented an simple airline booking system. The database scheme like this:
In the book flight API, the method will check if the flight is available and get the version number. Then the method will minus 1 from available_seats and check if the version number is same. If not the save() method will throw ObjectOptimisticLockingFailureException and catch by controller layer. The code is the service layer:
public Ticket bookFlight(Flight flight, long customerId) throws Exception {
Flight returnedFlight = flightRepository.findFlightByFlightNumberAndFlightDate(flight.getFlightNumber(), flight.getFlightDate());
if (validFlightIsAvailable(returnedFlight) == false) {
// A valid flight ticket id should greater than 0.
// Therefore, ticketId = 0 means the flight is full.
int fullFlightTicketId = 0;
log.info("Customer " + customerId + " failed to book the ticket because the flight is full.");
return new Ticket(fullFlightTicketId);
} else {
Ticket newTicket = new Ticket(customerId, returnedFlight.getFlightId(), returnedFlight.getFlightDate());
if (checkIsDuplicatedBooking(newTicket) == true){
log.error("The customer (" + customerId + ") already book the ticket for flight " + newTicket.getFlightId());
throw new ClientException("Customer already booked the ticket in the same flight", HttpStatus.BAD_REQUEST);
}
returnedFlight.setAvailableSeats(returnedFlight.getAvailableSeats() - 1);
Flight newFlight = flightRepository.save(returnedFlight);
if (newFlight == null) {
log.error("Update flight's available seats error. (flight Id: " +
returnedFlight.getFlightId() + "). Rollback all transactions.");
throw new ServerException("Unknown Server Exception.", HttpStatus.INTERNAL_SERVER_ERROR);
}
Ticket returnedTicket = ticketRepository.save(newTicket);
if (returnedTicket != null) {
log.info("Customer Id: " + customerId + " successfully booked the ticket in flight " + flight.getFlightId());
return returnedTicket;
} else {
log.error("Save new ticket into database error. (Customer Id: " + customerId + ", flight Id: " +
returnedFlight.getFlightId() + "). Rollback all transactions.");
throw new ServerException("Unknown Server Exception.", HttpStatus.INTERNAL_SERVER_ERROR);
}
}
}
The flight entity is :
#Entity
#Table(name = "flight")
public class Flight {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
#Column(name = "flight_id", nullable = false)
private Long flightId;
#Column(name = "flight_number", nullable = false)
private Long flightNumber;
#Column(name = "flight_date", nullable = false)
private Date flightDate;
#Column(name = "available_seats", nullable = false)
private Integer availableSeats;
#Column(name = "version")
#Version
private int version;
// Getter and Setter
I added a unit test for 50 users and use mockMvc to test the function.
#Test
void bookFlight_Concurrent_Success() throws Exception {
// Get all flight by Default flight number
List<Flight> availableFlights = flightService.getAllAvailableFlightsByFlightNumber(defaultFlights.get(0));
int flightIndex = 3;
int totalAvailableSeat = availableFlights.get(flightIndex).getAvailableSeats();
String requestJSON = "{\n" +
" \"flightDate\": \"" + availableFlights.get(flightIndex).getFlightDate().toString() + "\",\n" +
" \"flightNumber\": " + availableFlights.get(flightIndex).getFlightNumber() + "\n" +
"}";
// Set up builders
List<RequestBuilder> builders = new ArrayList<>();
for (int i = 0; i < defaultCustomerUsernames.size(); i++){
String jwt = getJWTByUsername(defaultCustomerUsernames.get(i), constants.CUSTOMER_USER_PASSWORD_0);
RequestBuilder builder = post("/bookFlight").header("Authorization", "Bearer " + jwt).
accept(MediaType.APPLICATION_JSON).content(requestJSON).contentType(MediaType.APPLICATION_JSON);
builders.add(builder);
}
// Start threads to book flight
List<Thread> threadList = new ArrayList<>();
for (int i = 0; i < defaultCustomerUsernames.size(); i++){
int finalI0 = i;
int finalI1 = i;
Thread thread = new Thread(){
public void run() {
try {
System.out.println(this.getName() + ": Start thread " + finalI1);
MvcResult result = mockMvc.perform(builders.get(finalI0)).andReturn();
System.out.println(this.getName() + ": Finish thread " + finalI1);
} catch (Exception e) {
e.printStackTrace();
}
}
};
thread.start();
threadList.add(thread);
}
try{
for (Thread thread: threadList){
thread.join();
}
}
catch (InterruptedException e){
e.printStackTrace();
}
// Verify the left ticket + booked ticket is full amount
int bookedTicketNum = ticketRepository.findTicketsByFlightId(availableFlights.get(flightIndex).getFlightId()).size();
int newAvailableTicket = flightRepository.findFlightByFlightId(availableFlights.get(flightIndex).getFlightId()).getAvailableSeats();
System.out.println("Booked Ticket Num: " + bookedTicketNum);
System.out.println("New Available Ticket: " + newAvailableTicket);
assertEquals(totalAvailableSeat, bookedTicketNum+newAvailableTicket);
}
beforeAll and afterAll are implemented to generate the default flight and customer user.
The test passed but
2020-09-22 21:34:00:667 ERROR [Thread-7] org.hibernate.engine.jdbc.batch.internal.BatchingBatch [130] : HHH000315: Exception executing batch [java.sql.BatchUpdateException: Deadlock found when trying to get lock; try restarting transaction], SQL: update flight set available_seats=?, flight_date=?, flight_number=?, version=? where flight_id=? and version=?
2020-09-22 21:34:00:668 ERROR [Thread-7] org.hibernate.engine.jdbc.spi.SqlExceptionHelper [142] : Deadlock found when trying to get lock; try restarting transaction
I used SHOW ENGINE INNODB STATUS to check deadlock log and get this:
=====================================
2020-09-22 21:36:50 0x700001f89000 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 17 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 1595 srv_active, 0 srv_shutdown, 22112 srv_idle
srv_master_thread log flush and writes: 0
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 2965
OS WAIT ARRAY INFO: signal count 44793
RW-shared spins 164384, rounds 166427, OS waits 1899
RW-excl spins 63944, rounds 93238, OS waits 183
RW-sx spins 28152, rounds 88279, OS waits 316
Spin rounds per wait: 1.01 RW-shared, 1.46 RW-excl, 3.14 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2020-09-22 21:34:00 0x70000156e000
*** (1) TRANSACTION:
TRANSACTION 34961, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 1
MySQL thread id 586, OS thread handle 123145342349312, query id 3508200 localhost 127.0.0.1 root updating
update flight set available_seats=149, flight_date='2020-12-14', flight_number=2, version=7 where flight_id=178360 and version=6
*** (1) HOLDS THE LOCK(S):
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 34961 lock mode S locks rec but not gap
Record lock, heap no 85 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002b8b8; asc ;;
1: len 6; hex 000000008889; asc ;;
2: len 7; hex 02000002091880; asc ;;
3: len 4; hex 80000002; asc ;;
4: len 3; hex 8fc98e; asc ;;
5: len 4; hex 80000095; asc ;;
6: len 4; hex 80000007; asc ;;
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 34961 lock_mode X locks rec but not gap waiting
Record lock, heap no 85 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002b8b8; asc ;;
1: len 6; hex 000000008889; asc ;;
2: len 7; hex 02000002091880; asc ;;
3: len 4; hex 80000002; asc ;;
4: len 3; hex 8fc98e; asc ;;
5: len 4; hex 80000095; asc ;;
6: len 4; hex 80000007; asc ;;
*** (2) TRANSACTION:
TRANSACTION 34959, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 1
MySQL thread id 591, OS thread handle 123145335984128, query id 3508204 localhost 127.0.0.1 root updating
update flight set available_seats=149, flight_date='2020-12-14', flight_number=2, version=7 where flight_id=178360 and version=6
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 34959 lock mode S locks rec but not gap
Record lock, heap no 85 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002b8b8; asc ;;
1: len 6; hex 000000008889; asc ;;
2: len 7; hex 02000002091880; asc ;;
3: len 4; hex 80000002; asc ;;
4: len 3; hex 8fc98e; asc ;;
5: len 4; hex 80000095; asc ;;
6: len 4; hex 80000007; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 34959 lock_mode X locks rec but not gap waiting
Record lock, heap no 85 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002b8b8; asc ;;
1: len 6; hex 000000008889; asc ;;
2: len 7; hex 02000002091880; asc ;;
3: len 4; hex 80000002; asc ;;
4: len 3; hex 8fc98e; asc ;;
5: len 4; hex 80000095; asc ;;
6: len 4; hex 80000007; asc ;;
*** WE ROLL BACK TRANSACTION (2)
------------
TRANSACTIONS
------------
Trx id counter 35089
Purge done for trx's n:o < 35089 undo n:o < 0 state: running but idle
History list length 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 422038541902072, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541902912, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541901232, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541897872, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541899552, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541898712, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541897032, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541896192, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541895352, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (read thread)
I/O thread 4 state: waiting for i/o request (read thread)
I/O thread 5 state: waiting for i/o request (read thread)
I/O thread 6 state: waiting for i/o request (write thread)
I/O thread 7 state: waiting for i/o request (write thread)
I/O thread 8 state: waiting for i/o request (write thread)
I/O thread 9 state: waiting for i/o request (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 1
1276 OS file reads, 982454 OS file writes, 737797 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 2 buffer(s)
Hash table size 34679, node heap has 2 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 4 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 1793856320
Log buffer assigned up to 1793856320
Log buffer completed up to 1793856320
Log written up to 1793856320
Log flushed up to 1793856320
Added dirty pages up to 1793856320
Pages flushed up to 1793856320
Last checkpoint at 1793856320
937447 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 137363456
Dictionary memory allocated 476872
Buffer pool size 8191
Free buffers 5484
Database pages 2694
Old database pages 982
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 4, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1241, created 1453, written 34817
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2694, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=2483, Main thread ID=0x70000177a000 , state=sleeping
Number of rows inserted 2692764, updated 1168, deleted 1168363, read 1256843
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
Number of system rows inserted 22, updated 377, deleted 20, read 13121
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
I think two update transaction cause this issue.
I tried to manual debug, and still can use other tools update the row after Flight newFlight = flightRepository.save(returnedFlight);
But, I cannot update the row after Ticket returnedTicket = ticketRepository.save(newTicket);
The deadlock log:
=====================================
2020-09-22 21:41:23 0x700001f89000 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 29 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 1597 srv_active, 0 srv_shutdown, 22382 srv_idle
srv_master_thread log flush and writes: 0
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 2965
OS WAIT ARRAY INFO: signal count 44793
RW-shared spins 164384, rounds 166427, OS waits 1899
RW-excl spins 63944, rounds 93238, OS waits 183
RW-sx spins 28152, rounds 88279, OS waits 316
Spin rounds per wait: 1.01 RW-shared, 1.46 RW-excl, 3.14 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2020-09-22 21:41:17 0x70000156e000
*** (1) TRANSACTION:
TRANSACTION 35092, ACTIVE 24 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 1136, 1 row lock(s)
MySQL thread id 8, OS thread handle 123145333862400, query id 3508825 localhost 127.0.0.1 root updating
/* ApplicationName=DataGrip 2020.2.2 */ UPDATE airline_booking_system.flight t SET t.version = 12 WHERE t.flight_id = 173431
*** (1) HOLDS THE LOCK(S):
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 35092 lock_mode X locks rec but not gap waiting
Record lock, heap no 4 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002a577; asc w;;
1: len 6; hex 000000008911; asc ;;
2: len 7; hex 020000016c0c77; asc l w;;
3: len 4; hex 80000373; asc s;;
4: len 3; hex 8fc938; asc 8;;
5: len 4; hex 80000096; asc ;;
6: len 4; hex 8000000b; asc ;;
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 35092 lock_mode X locks rec but not gap waiting
Record lock, heap no 4 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002a577; asc w;;
1: len 6; hex 000000008911; asc ;;
2: len 7; hex 020000016c0c77; asc l w;;
3: len 4; hex 80000373; asc s;;
4: len 3; hex 8fc938; asc 8;;
5: len 4; hex 80000096; asc ;;
6: len 4; hex 8000000b; asc ;;
*** (2) TRANSACTION:
TRANSACTION 35091, ACTIVE 41 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 1
MySQL thread id 599, OS thread handle 123145336287232, query id 3508827 localhost 127.0.0.1 root updating
update flight set available_seats=149, flight_date='2020-09-24', flight_number=883, version=11 where flight_id=173431 and version=10
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 35091 lock mode S locks rec but not gap
Record lock, heap no 4 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002a577; asc w;;
1: len 6; hex 000000008911; asc ;;
2: len 7; hex 020000016c0c77; asc l w;;
3: len 4; hex 80000373; asc s;;
4: len 3; hex 8fc938; asc 8;;
5: len 4; hex 80000096; asc ;;
6: len 4; hex 8000000b; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 17 page no 4 n bits 480 index PRIMARY of table `airline_booking_system`.`flight` trx id 35091 lock_mode X locks rec but not gap waiting
Record lock, heap no 4 PHYSICAL RECORD: n_fields 7; compact format; info bits 128
0: len 4; hex 8002a577; asc w;;
1: len 6; hex 000000008911; asc ;;
2: len 7; hex 020000016c0c77; asc l w;;
3: len 4; hex 80000373; asc s;;
4: len 3; hex 8fc938; asc 8;;
5: len 4; hex 80000096; asc ;;
6: len 4; hex 8000000b; asc ;;
*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------
Trx id counter 35093
Purge done for trx's n:o < 35091 undo n:o < 0 state: running but idle
History list length 1
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 422038541900392, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541902072, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541902912, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541901232, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541897872, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541899552, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541898712, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541897032, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541896192, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 422038541895352, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (read thread)
I/O thread 4 state: waiting for i/o request (read thread)
I/O thread 5 state: waiting for i/o request (read thread)
I/O thread 6 state: waiting for i/o request (write thread)
I/O thread 7 state: waiting for i/o request (write thread)
I/O thread 8 state: waiting for i/o request (write thread)
I/O thread 9 state: waiting for i/o request (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 1
1276 OS file reads, 982489 OS file writes, 737827 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.24 writes/s, 0.17 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 2 buffer(s)
Hash table size 34679, node heap has 2 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 4 buffer(s)
0.03 hash searches/s, 0.10 non-hash searches/s
---
LOG
---
Log sequence number 1793857089
Log buffer assigned up to 1793857089
Log buffer completed up to 1793857089
Log written up to 1793857089
Log flushed up to 1793857089
Added dirty pages up to 1793857089
Pages flushed up to 1793857089
Last checkpoint at 1793857089
937456 log i/o's done, 0.03 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 137363456
Dictionary memory allocated 476872
Buffer pool size 8191
Free buffers 5484
Database pages 2694
Old database pages 982
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 4, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1241, created 1453, written 34832
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2694, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=2483, Main thread ID=0x70000177a000 , state=sleeping
Number of rows inserted 2692765, updated 1169, deleted 1168363, read 1256855
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.03 reads/s
Number of system rows inserted 22, updated 377, deleted 20, read 13205
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
If anyone can help me? I tried Pessimistic Lock and no issue.

How to solve Deadlock in query with sub select?

When I try to make UPDATE with subselect I catch deadlocks.
I caught this deadlock in my application I am using Hibernate for get data from DB and in my log, I see next:
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
When I removed subselect from a query it`s solved deadlock but in my case these subqueries are necessary.
Nested update with select deadlock
https://www.percona.com/blog/2017/09/25/avoid-shared-locks-from-subqueries-when-possible/
But these solutions don't solve my problem.
What I do wrong?
Maby someone had a similar problem and can help.
It is my UPDATE with subselect:
UPDATE mediaplan m
SET m.startDate = IFNULL((SELECT p.blockStartTime
FROM placement p
WHERE p.mediaplan_id = m.id AND p.deleted = FALSE
ORDER BY p.blockStartTime ASC
LIMIT 0, 1), NULL),
m.endDate = IFNULL((SELECT p.blockStartTime
FROM placement p
WHERE p.mediaplan_id = m.id AND p.deleted = FALSE
ORDER BY p.blockStartTime DESC
LIMIT 0, 1), NULL)
WHERE m.id IN (208396);
Deadlock log (SHOW ENGINE INNODB STATUS;)
=====================================
2018-03-28 15:00:52 0x700007086000 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 38 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 124 srv_active, 0 srv_shutdown, 9865 srv_idle
srv_master_thread log flush and writes: 9989
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 3037
OS WAIT ARRAY INFO: signal count 5042
RW-shared spins 0, rounds 7370, OS waits 1740
RW-excl spins 0, rounds 47629, OS waits 217
RW-sx spins 268, rounds 7782, OS waits 245
Spin rounds per wait: 7370.00 RW-shared, 47629.00 RW-excl, 29.04 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2018-03-28 15:00:32 0x700006f32000
*** (1) TRANSACTION:
TRANSACTION 1767639, ACTIVE 0 sec starting index read
mysql tables in use 3, locked 3
LOCK WAIT 6 lock struct(s), heap size 1136, 4 row lock(s), undo log entries 2
MySQL thread id 1105, OS thread handle 123145416400896, query id 17464 localhost 127.0.0.1 romexdev updating
UPDATE mediaplan m SET m.startDate = IFNULL( (SELECT p.blockStartTime FROM placement p WHERE p.mediaplan_id = m.id AND p.deleted = FALSE ORDER BY p.blockStartTime ASC LIMIT 0, 1), NULL ), m.endDate = IFNULL( (SELECT p.blockStartTime FROM placement p WHERE p.mediaplan_id = m.id AND p.deleted = FALSE ORDER BY p.blockStartTime DESC LIMIT 0, 1), NULL ) WHERE m.id IN (-1,208396)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1658 page no 541 n bits 320 index PRIMARY of table `openmarket`.`mediaplan` trx id 1767639 lock_mode X locks rec but not gap waiting
Record lock, heap no 16 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 4; hex 80032e0c; asc . ;;
1: len 6; hex 0000001af8d3; asc ;;
2: len 7; hex 7e000009480854; asc ~ H T;;
3: len 5; hex 99a006fc80; asc ;;
4: len 5; hex 999f748c00; asc t ;;
5: len 4; hex 8000027e; asc ~;;
6: SQL NULL;
7: len 1; hex 00; asc ;;
8: len 5; hex 999f6cc39f; asc l ;;
9: len 5; hex 999f6cc39f; asc l ;;
10: len 4; hex 800000a7; asc ;;
*** (2) TRANSACTION:
TRANSACTION 1767637, ACTIVE 0 sec fetching rows
mysql tables in use 3, locked 3
37 lock struct(s), heap size 3520, 900 row lock(s), undo log entries 2
MySQL thread id 1107, OS thread handle 123145418907648, query id 17455 localhost 127.0.0.1 romexdev Creating sort index
UPDATE mediaplan m SET m.startDate = IFNULL( (SELECT p.blockStartTime FROM placement p WHERE p.mediaplan_id = m.id AND p.deleted = FALSE ORDER BY p.blockStartTime ASC LIMIT 0, 1), NULL ), m.endDate = IFNULL( (SELECT p.blockStartTime FROM placement p WHERE p.mediaplan_id = m.id AND p.deleted = FALSE ORDER BY p.blockStartTime DESC LIMIT 0, 1), NULL ) WHERE m.id IN (-1,208396)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1658 page no 541 n bits 320 index PRIMARY of table `openmarket`.`mediaplan` trx id 1767637 lock_mode X locks rec but not gap
Record lock, heap no 16 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 4; hex 80032e0c; asc . ;;
1: len 6; hex 0000001af8d3; asc ;;
2: len 7; hex 7e000009480854; asc ~ H T;;
3: len 5; hex 99a006fc80; asc ;;
4: len 5; hex 999f748c00; asc t ;;
5: len 4; hex 8000027e; asc ~;;
6: SQL NULL;
7: len 1; hex 00; asc ;;
8: len 5; hex 999f6cc39f; asc l ;;
9: len 5; hex 999f6cc39f; asc l ;;
10: len 4; hex 800000a7; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1752 page no 435722 n bits 240 index PRIMARY of table `openmarket`.`placement` trx id 1767637 lock mode S locks rec but not gap waiting
Record lock, heap no 144 PHYSICAL RECORD: n_fields 31; compact format; info bits 0
0: len 4; hex 80c66779; asc gy;;
1: len 5; hex 999fa60000; asc ;;
2: len 6; hex 0000001af8d7; asc ;;
3: len 7; hex 210000c0160979; asc ! y;;
4: len 5; hex 999f78c6f6; asc x ;;
5: len 1; hex 00; asc ;;
6: len 1; hex 83; asc ;;
7: SQL NULL;
8: len 4; hex 88cb341e; asc 4 ;;
9: len 4; hex 80032e0c; asc . ;;
10: len 4; hex 80033ce3; asc < ;;
11: len 1; hex 01; asc ;;
12: len 5; hex 999fa62440; asc $#;;
13: SQL NULL;
14: SQL NULL;
15: SQL NULL;
16: SQL NULL;
17: SQL NULL;
18: SQL NULL;
19: len 1; hex 00; asc ;;
20: len 5; hex 999f78c6f6; asc x ;;
21: len 5; hex 999f78f020; asc x ;;
22: len 4; hex 80000001; asc ;;
23: SQL NULL;
24: len 1; hex 01; asc ;;
25: len 1; hex 01; asc ;;
26: len 1; hex 00; asc ;;
27: len 1; hex 00; asc ;;
28: len 4; hex 8002afaf; asc ;;
29: len 4; hex 80000012; asc ;;
30: SQL NULL;
*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------
Trx id counter 1767643
Purge done for trx's n:o < 1767643 undo n:o < 0 state: running but idle
History list length 31
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 281479519987928, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519987024, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519986120, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519985216, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519984312, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519989736, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519988832, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479519983408, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (read thread)
I/O thread 4 state: waiting for i/o request (read thread)
I/O thread 5 state: waiting for i/o request (read thread)
I/O thread 6 state: waiting for i/o request (write thread)
I/O thread 7 state: waiting for i/o request (write thread)
I/O thread 8 state: waiting for i/o request (write thread)
I/O thread 9 state: waiting for i/o request (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
508449 OS file reads, 49980 OS file writes, 15796 OS fsyncs
0.24 reads/s, 16384 avg bytes/read, 2.58 writes/s, 0.76 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 3092, seg size 3094, 84 merges
merged operations:
insert 134, delete mark 3, delete 1
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 34673, node heap has 4 buffer(s)
Hash table size 34673, node heap has 24 buffer(s)
Hash table size 34673, node heap has 62 buffer(s)
Hash table size 34673, node heap has 201 buffer(s)
Hash table size 34673, node heap has 73 buffer(s)
Hash table size 34673, node heap has 2 buffer(s)
Hash table size 34673, node heap has 1 buffer(s)
Hash table size 34673, node heap has 2 buffer(s)
133.47 hash searches/s, 26.13 non-hash searches/s
---
LOG
---
Log sequence number 212087243581
Log flushed up to 212087243581
Pages flushed up to 212087243581
Last checkpoint at 212087243572
0 pending log flushes, 0 pending chkp writes
678 log i/o's done, 0.39 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 137428992
Dictionary memory allocated 5812878
Buffer pool size 8191
Free buffers 1024
Database pages 6798
Old database pages 2489
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 24521, not young 13076464
0.00 youngs/s, 1.39 non-youngs/s
Pages read 504777, created 29725, written 31405
0.24 reads/s, 0.00 creates/s, 2.11 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 3 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 6798, unzip_LRU len: 0
I/O sum[130]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=36398, Main thread ID=123145409064960, state: sleeping
Number of rows inserted 49479, updated 123, deleted 0, read 16253106
0.08 inserts/s, 0.16 updates/s, 0.00 deletes/s, 276.97 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
Try
BEGIN;
SELECT #start := ... LIMIT 1;
SELECT #end := ... LIMIT 1;
UPDATE ...
SET startDate = COALESCE(startDate, #start),
endDate = COALESCE(startDate, #end)
WHERE ...
COMMIT;

MySQL excessive memory usage

I have an installation of MySQL which allocates a lot more memory than I expected. I'd like to understand where it has gone to fix the root cause.
To estimate the RAM usage I use this formula:
key_buffer_size + query_cache_size + innodb_buffer_pool_size + innodb_additional_mem_pool_size + innodb_log_buffer_size + Max_used_connections * (read_buffer_size + read_rnd_buffer_size + sort_buffer_size+ join_buffer_size+ binlog_cache_size + thread_stack + tmp_table_size)
This formula produces around 5.3 GB of estimated allocation.
Instead, the RAM used by MySQL continues to grow and, after a couple of days of activity, it goes well over 9 GB (writing heavily into swap).
What am I forgetting? How can I understand who is eating the remaining RAM?
A few more pieces of information.
I'm running MySQL CE 5.6.17 on RHEL6.3 64 bits. My server has 6 GB RAM and 8GB swap space.
I have constantly around 150 alive connections to MySQL but I'm keeping max_connections to higher values to accommodate peaks and foresee some more traffic on this machine.
I'm only using InnoDB.
My application uses pooled connections to the DB. The producers open connections and release them to the pool when they are done. I have measured that I always have several parallel requests, and pooling speeds up a little bit connections creation (so faster queries). The pool (DBCP) auto-tunes a reasonable number of alive connections and it's usually around 100-150.
I have already tried:
killing the connections does not free RAM
FLUSH TABLES does not free RAM
Following my config:
---- my.cnf -----
[client]
port = 3306
socket = /var/lib/mysql/mysql.sock
default-character-set=utf8
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-external-locking
key_buffer_size = 64M
max_allowed_packet = 1M
table_open_cache = 256
key_buffer_size = 128M
sort_buffer_size = 524288
read_buffer_size = 131072
read_rnd_buffer_size = 524288
myisam_sort_buffer_size = 524288
tmp_table_size=10M
performance_schema=0
thread_cache_size = 50
query_cache_type = OFF
query_cache_size = 0
server-id = 1
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
innodb_data_home_dir = "/data/mysqldata"
innodb_log_group_home_dir = "/data1/mysqllog"
innodb_buffer_pool_size = 3000M
innodb_buffer_pool_instances = 3
innodb_log_file_size = 750M
innodb_flush_log_at_trx_commit = 2
datadir = "/data/mysqldata"
innodb_flush_method=O_DIRECT
max_connections = 1650
slow_query_log = 1
slow_query_log_file = slow.log
long_query_time = 10
autocommit=1
init-connect='SET NAMES utf8'
character-set-server = utf8
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
default-character-set=utf8
[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
---------
This is the memory allocated to mysql after few days of activity:
[root#G-VelocityDB ~]# ps aux | grep mysql
root 1963 0.0 0.0 108332 1492 ? S May09 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysqldata --pid-file=/data/mysqldata/G-VelocityDB.pid
mysql 2583 80.4 88.3 9816560 8949704 ? Sl May09 3538:34 /usr/sbin/mysqld --basedir=/usr --datadir=/data/mysqldata --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/data/mysqldata/G-VelocityDB.err --pid-file=/data/mysqldata/G-VelocityDB.pid --socket=/var/lib/mysql/mysql.sock --port=3306
root 21450 0.0 0.0 103244 852 pts/1 S+ 08:51 0:00 grep mysql
# cat /proc/2583/status
Name: mysqld
State: S (sleeping)
Tgid: 2583
Pid: 2583
PPid: 1963
TracerPid: 0
Uid: 496 496 496 496
Gid: 493 493 493 493
Utrace: 0
FDSize: 512
Groups: 493
VmPeak: 9882096 kB
VmSize: 9816560 kB
VmLck: 0 kB
VmHWM: 8950684 kB
VmRSS: 8949652 kB
VmData: 9771648 kB
VmStk: 88 kB
VmExe: 11684 kB
VmLib: 3928 kB
VmPTE: 17996 kB
VmSwap: 0 kB
Threads: 161
SigQ: 0/78996
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000087007
SigIgn: 0000000000001006
SigCgt: 00000001800066e9
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed: 3
Cpus_allowed_list: 0-1
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 15496
nonvoluntary_ctxt_switches: 337
Some more info on my server:
GLOBAL STATUS:
Aborted_clients............ 237
Aborted_connects............. 10
Binlog_cache_disk_use.......... 0
Binlog_cache_use............. 0
Binlog_stmt_cache_disk_use......... 0
Binlog_stmt_cache_use.......... 0
Bytes_received............. 4194923507
Bytes_sent............. 11895871264
Com_admin_commands........... 455
Com_assign_to_keycache........... 0
Com_alter_db............. 0
Com_alter_db_upgrade........... 0
Com_alter_event............ 0
Com_alter_function........... 0
Com_alter_procedure.......... 0
Com_alter_server............. 0
Com_alter_table............ 0
Com_alter_tablespace........... 0
Com_alter_user............. 0
Com_analyze............ 0
Com_begin.............. 2
Com_binlog............. 0
Com_call_procedure........... 4868660
Com_change_db............ 8
Com_change_master............ 0
Com_check.............. 0
Com_checksum............. 0
Com_create_db............ 0
Com_create_event............. 0
Com_create_function.......... 0
Com_create_index............. 0
Com_create_procedure........... 0
Com_create_server............ 0
Com_create_table............. 0
Com_create_trigger........... 0
Com_create_udf............. 0
Com_create_user............ 0
Com_create_view............ 0
Com_dealloc_sql............ 0
Com_delete............. 19742
Com_delete_multi............. 0
Com_do............... 0
Com_drop_db............ 0
Com_drop_event............. 0
Com_drop_function............ 0
Com_drop_index............. 0
Com_drop_procedure........... 0
Com_drop_server............ 0
Com_drop_table............. 0
Com_drop_trigger............. 0
Com_drop_user............ 0
Com_drop_view............ 0
Com_empty_query............ 0
Com_execute_sql............ 0
Com_flush.............. 0
Com_get_diagnostics.......... 0
Com_grant.............. 0
Com_ha_close............. 0
Com_ha_open............ 0
Com_ha_read............ 0
Com_help............... 57
Com_insert............. 7149329
Com_insert_select............ 44
Com_install_plugin........... 0
Com_kill............... 0
Com_load............... 0
Com_lock_tables............ 0
Com_optimize............. 0
Com_preload_keys............. 0
Com_prepare_sql............ 0
Com_purge.............. 0
Com_purge_before_date.......... 0
Com_release_savepoint.......... 0
Com_rename_table............. 0
Com_rename_user............ 0
Com_repair............. 0
Com_replace............ 0
Com_replace_select........... 0
Com_reset.............. 0
Com_resignal............. 0
Com_revoke............. 0
Com_revoke_all............. 0
Com_rollback............. 6366209
Com_rollback_to_savepoint.......... 0
Com_savepoint............ 0
Com_select............. 16223159
Com_set_option............. 13471816
Com_signal............. 0
Com_show_binlog_events........... 0
Com_show_binlogs............. 0
Com_show_charsets............ 0
Com_show_collations.......... 632
Com_show_create_db........... 0
Com_show_create_event.......... 0
Com_show_create_func........... 0
Com_show_create_proc........... 0
Com_show_create_table.......... 0
Com_show_create_trigger.......... 0
Com_show_databases........... 21
Com_show_engine_logs........... 0
Com_show_engine_mutex.......... 0
Com_show_engine_status........... 0
Com_show_events............ 0
Com_show_errors............ 0
Com_show_fields............ 101
Com_show_function_code........... 0
Com_show_function_status........... 3
Com_show_grants............ 0
Com_show_keys............ 9
Com_show_master_status........... 0
Com_show_open_tables........... 0
Com_show_plugins............. 2
Com_show_privileges.......... 0
Com_show_procedure_code.......... 0
Com_show_procedure_status.......... 3
Com_show_processlist........... 0
Com_show_profile............. 0
Com_show_profiles............ 0
Com_show_relaylog_events........... 0
Com_show_slave_hosts........... 0
Com_show_slave_status.......... 2
Com_show_status............ 805
Com_show_storage_engines........... 0
Com_show_table_status.......... 0
Com_show_tables............ 13
Com_show_triggers............ 0
Com_show_variables........... 667
Com_show_warnings............ 0
Com_slave_start............ 0
Com_slave_stop............. 0
Com_stmt_close............. 0
Com_stmt_execute............. 0
Com_stmt_fetch............. 0
Com_stmt_prepare............. 0
Com_stmt_reprepare........... 0
Com_stmt_reset............. 0
Com_stmt_send_long_data.......... 0
Com_truncate............. 0
Com_uninstall_plugin........... 0
Com_unlock_tables............ 0
Com_update............. 168248
Com_update_multi............. 0
Com_xa_commit............ 0
Com_xa_end............. 0
Com_xa_prepare............. 0
Com_xa_recover............. 0
Com_xa_rollback............ 0
Com_xa_start............. 0
Compression............ OFF
Connection_errors_accept........... 0
Connection_errors_internal......... 0
Connection_errors_max_connections........ 0
Connection_errors_peer_address......... 0
Connection_errors_select........... 0
Connection_errors_tcpwrap.......... 0
Connections............ 670
Created_tmp_disk_tables.......... 838
Created_tmp_files............ 6
Created_tmp_tables........... 5214
Delayed_errors............. 0
Delayed_insert_threads........... 0
Delayed_writes............. 0
Flush_commands............. 1
Handler_commit............. 30081211
Handler_delete............. 6300384
Handler_discover............. 0
Handler_external_lock.......... 67018766
Handler_mrr_init............. 0
Handler_prepare............ 0
Handler_read_first........... 11247573
Handler_read_key............. 22140615
Handler_read_last............ 0
Handler_read_next............ 1099116268
Handler_read_prev............ 26294847
Handler_read_rnd............. 38722
Handler_read_rnd_next.......... 22705321
Handler_rollback............. 60
Handler_savepoint............ 0
Handler_savepoint_rollback......... 0
Handler_update............. 167974
Handler_write............ 7852897
Innodb_buffer_pool_dump_status......... not started
Innodb_buffer_pool_load_status......... not started
Innodb_buffer_pool_pages_data........ 186890
Innodb_buffer_pool_bytes_data........ 3062005760
Innodb_buffer_pool_pages_dirty......... 6539
Innodb_buffer_pool_bytes_dirty......... 107134976
Innodb_buffer_pool_pages_flushed......... 584944
Innodb_buffer_pool_pages_free........ 3072
Innodb_buffer_pool_pages_misc........ 2035
Innodb_buffer_pool_pages_total......... 191997
Innodb_buffer_pool_read_ahead_rnd........ 0
Innodb_buffer_pool_read_ahead........ 129315
Innodb_buffer_pool_read_ahead_evicted...... 0
Innodb_buffer_pool_read_requests......... 531706692
Innodb_buffer_pool_reads........... 215710
Innodb_buffer_pool_wait_free......... 0
Innodb_buffer_pool_write_requests........ 84850509
Innodb_data_fsyncs........... 445789
Innodb_data_pending_fsyncs......... 0
Innodb_data_pending_reads.......... 0
Innodb_data_pending_writes......... 0
Innodb_data_read............. 5930536960
Innodb_data_reads............ 379282
Innodb_data_writes........... 3707043
Innodb_data_written.......... 24355757056
Innodb_dblwr_pages_written......... 584944
Innodb_dblwr_writes.......... 16463
Innodb_have_atomic_builtins........ ON
Innodb_log_waits............. 0
Innodb_log_write_requests.......... 8093999
Innodb_log_writes............ 3091626
Innodb_os_log_fsyncs........... 168887
Innodb_os_log_pending_fsyncs......... 0
Innodb_os_log_pending_writes......... 0
Innodb_os_log_written.......... 5187414528
Innodb_page_size............. 16384
Innodb_pages_created........... 76352
Innodb_pages_read............ 358104
Innodb_pages_written........... 584944
Innodb_row_lock_current_waits........ 0
Innodb_row_lock_time........... 17326
Innodb_row_lock_time_avg........... 18
Innodb_row_lock_time_max........... 1027
Innodb_row_lock_waits.......... 946
Innodb_rows_deleted.......... 6300384
Innodb_rows_inserted........... 7149273
Innodb_rows_read............. 1140478060
Innodb_rows_updated.......... 164739
Innodb_num_open_files.......... 300
Innodb_truncated_status_writes......... 0
Innodb_available_undo_logs......... 128
Key_blocks_not_flushed........... 0
Key_blocks_unused............ 107146
Key_blocks_used............ 25
Key_read_requests............ 811
Key_reads.............. 25
Key_write_requests........... 0
Key_writes............. 0
Last_query_cost............ 0.000000
Last_query_partial_plans........... 0
Max_used_connections........... 172
Not_flushed_delayed_rows........... 0
Open_files............. 11
Open_streams............. 0
Open_table_definitions........... 69
Open_tables............ 256
Opened_files............. 3701
Opened_table_definitions........... 69
Opened_tables............ 308
Prepared_stmt_count.......... 0
Queries.............. 54874967
Questions.............. 49932690
Select_full_join............. 1196
Select_full_range_join........... 0
Select_range............. 494248
Select_range_check........... 0
Select_scan............ 9551157
Slow_launch_threads.......... 0
Slow_queries............. 63
Sort_merge_passes............ 0
Sort_range............. 1
Sort_rows.............. 45942
Sort_scan.............. 2550
Table_locks_immediate.......... 26010899
Table_locks_waited........... 0
Table_open_cache_hits.......... 24039437
Table_open_cache_misses.......... 308
Table_open_cache_overflows......... 45
Tc_log_max_pages_used.......... 0
Tc_log_page_size............. 0
Tc_log_page_waits............ 0
Threads_cached............. 47
Threads_connected............ 102
Threads_created............ 282
Threads_running............ 1
Uptime............... 171023
Uptime_since_flush_status.......... 171023
Com_commit............. 6536509
SHOW ENGINE INNODB STATUS;
Per second averages calculated from the last 17 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 268650 srv_active, 0 srv_shutdown, 12135 srv_idle
srv_master_thread log flush and writes: 280785
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 1688081
OS WAIT ARRAY INFO: signal count 957822
Mutex spin waits 1873319, rounds 44188913, OS waits 1295453
RW-shared spins 469687, rounds 12569665, OS waits 334361
RW-excl spins 35863, rounds 1100713, OS waits 24644
Spin rounds per wait: 23.59 mutex, 26.76 RW-shared, 30.69 RW-excl
------------------------
LATEST DETECTED DEADLOCK
------------------------
2014-05-07 17:22:56 7f0d94a1a700
*** (1) TRANSACTION:
TRANSACTION 1119978000, ACTIVE 0 sec starting index read
mysql tables in use 10, locked 10
LOCK WAIT 7 lock struct(s), heap size 1184, 2 row lock(s)
MySQL thread id 69, OS thread handle 0x7f0d91c0d700, query id 73382628 g-velocity.abodata.domain 10.0.2.180 platone updating
DELETE FROM PacketDay PARTITION(p1730,p1731,p1732,p1733,p1734,p1735,p1736,p1737,p1738,p1739) WHERE XXXX=4 AND StartTime<1399388476900 LIMIT 50000
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 5831 page no 27 n bits 224 index `PRIMARY` of table `ViewDb`.`PacketDay` /* Partition `p1732` */ trx id 1119978000 lock_mode X locks rec but not gap waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 17; compact format; info bits 32
0: len 8; hex 80000145cf7ab51d; asc E z ;;
1: len 4; hex 80000e94; asc ;;
2: len 6; hex 000042c1820e; asc B ;;
3: len 7; hex 65000480151812; asc e ;;
4: len 4; hex 80000005; asc ;;
5: len 8; hex 80000145cf7ab51d; asc E z ;;
6: len 8; hex 8000000000000000; asc ;;
7: len 4; hex 80000004; asc ;;
8: len 4; hex 80000003; asc ;;
9: SQL NULL;
10: len 8; hex 0000000000001c40; asc #;;
11: len 0; hex ; asc ;;
12: len 1; hex 30; asc 0;;
13: len 2; hex 8001; asc ;;
14: len 4; hex 80000000; asc ;;
15: len 8; hex 80000145cf7ab3b6; asc E z ;;
16: len 24; hex 5b303d4f66662c20313d52696768742c20323d4c6566745d; asc [0=Off, 1=Right, 2=Left];;
*** (2) TRANSACTION:
TRANSACTION 1119977998, ACTIVE 0 sec updating or deleting
mysql tables in use 10, locked 10
32 lock struct(s), heap size 6544, 6 row lock(s), undo log entries 1
MySQL thread id 36, OS thread handle 0x7f0d94a1a700, query id 73382652 g-velocity.abodata.domain 10.0.2.180 platone updating
DELETE FROM PacketDay PARTITION(p1730,p1731,p1732,p1733,p1734,p1735,p1736,p1737,p1738,p1739) WHERE XXXX=5 AND StartTime<1399388476871 LIMIT 50000
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 5831 page no 27 n bits 224 index `PRIMARY` of table `ViewDb`.`PacketDay` /* Partition `p1732` */ trx id 1119977998 lock_mode X locks rec but not gap
Record lock, heap no 2 PHYSICAL RECORD: n_fields 17; compact format; info bits 32
0: len 8; hex 80000145cf7ab51d; asc E z ;;
1: len 4; hex 80000e94; asc ;;
2: len 6; hex 000042c1820e; asc B ;;
3: len 7; hex 65000480151812; asc e ;;
4: len 4; hex 80000005; asc ;;
5: len 8; hex 80000145cf7ab51d; asc E z ;;
6: len 8; hex 8000000000000000; asc ;;
7: len 4; hex 80000004; asc ;;
8: len 4; hex 80000003; asc ;;
9: SQL NULL;
10: len 8; hex 0000000000001c40; asc #;;
11: len 0; hex ; asc ;;
12: len 1; hex 30; asc 0;;
13: len 2; hex 8001; asc ;;
14: len 4; hex 80000000; asc ;;
15: len 8; hex 80000145cf7ab3b6; asc E z ;;
16: len 24; hex 5b303d4f66662c20313d52696768742c20323d4c6566745d; asc [0=Off, 1=Right, 2=Left];;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 5831 page no 28 n bits 824 index `idx_XXXX_YYYY` of table `AAAA`.`PacketDay` /* Partition `p1732` */ trx id 1119977998 lock_mode X locks rec but not gap waiting
Record lock, heap no 686 PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 4; hex 80000005; asc ;;
1: len 8; hex 80000145cf7ab51d; asc E z ;;
2: len 4; hex 80000e94; asc ;;
*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------
Trx id counter 1136364858
Purge done for trx's n:o < 1136364106 undo n:o < 0 state: running but idle
History list length 984
LIST OF TRANSACTIONS FOR EACH SESSION:
[...]
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
1320822 OS file reads, 8503653 OS file writes, 2583777 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.12 writes/s, 0.06 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 67353, seg size 67355, 5035 merges
merged operations:
insert 65890, delete mark 110773, delete 319
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 6225523, node heap has 1792 buffer(s)
0.06 hash searches/s, 0.94 non-hash searches/s
---
LOG
---
Log sequence number 291212776237
Log flushed up to 291212776237
Pages flushed up to 291114806316
Last checkpoint at 291114806316
0 pending log writes, 0 pending chkp writes
4968792 log i/o's done, 0.12 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 3219456000; in additional pool allocated 0
Dictionary memory allocated 36167844
Buffer pool size 191997
Free buffers 3072
Database pages 187133
Old database pages 69022
Modified db pages 8213
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 653753, not young 38903623
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1310744, created 130762, written 3414584
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 187133, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 63999
Free buffers 1024
Database pages 62402
Old database pages 23015
Modified db pages 2482
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 197235, not young 11921942
0.00 youngs/s, 0.00 non-youngs/s
Pages read 423522, created 40807, written 1078968
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 62402, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 63999
Free buffers 1024
Database pages 62366
Old database pages 23006
Modified db pages 2909
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 219800, not young 13508215
0.00 youngs/s, 0.00 non-youngs/s
Pages read 439314, created 44996, written 1162507
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 62366, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 63999
Free buffers 1024
Database pages 62365
Old database pages 23001
Modified db pages 2822
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 236718, not young 13473466
0.00 youngs/s, 0.00 non-youngs/s
Pages read 447908, created 44959, written 1173109
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 62365, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
7 read views open inside InnoDB
Main thread process no. 5965, id 139696323659520, state: sleeping
Number of rows inserted 13664081, updated 230718, deleted 8001095, read 31517028476
0.06 inserts/s, 0.06 updates/s, 0.00 deletes/s, 43.59 reads/s
select sum(data_length+index_length) from information_schema.tables where engine='memory';
Result: '0'
As suggested, I have run with and without partitioning enabled on my server (I have 5 tables with 2000 partitions each), and seems that disabling partitioning helps in some way. But still I have not clear the reason, and I cannot eliminate partitioning at all.
As suggested, I have run some basic test running mysql under valgrind for leak detection: http://pastebin.com/W3ZMAGY4
And massif: http://pastebin.com/LweQHn3M
These tests are executed with MySQL default configuration.
If I'm reading the output correctly, no relevant leak is found and I have a huge memory consumption per connection (500 MB each connection).
Is that correct? Any idea why it grows so much?
The InnoDB data dictionary can grow without bounds as you open many tables, beyond innodb_additional_mem_pool_size, and it often does grow huge if you have thousands of tables. This would be independent of the number of connections.
I've seen other people report that MySQL 5.6 has a lot of memory usage, but we haven't tracked it down definitively. One would have to run mysqld under valgrind to track memory growth.
MySQL 5.7 is still under development, but they are creating new PERFORMANCE_SCHEMA instruments to track memory usage.
If the memory reported by the InnoDB data dictionary alone doesn't account for the memory growth, I suggest reading the following bug reports to see if they apply in your case:
Bug #72322 Query to I_S.tables and I_S.columns leads to huge memory usage
Bug #68514 5.6 huge memory usage with max_connections parameter
Bug #57480 Memory leak when having 256+ tables

App hangs due to unknown long mysql transactions

I have a Rails application running on mysql 5.6. 1-2 times a day the application hangs and restarting the server gets things back running.
In my logs, it looks like I get LOCK WAIT TIMEOUTS, The problem looks like it is in MySQL, in a SHOW ENGINE INNODB STATUS\G this is usually what we see:
---TRANSACTION 13339315173, ACTIVE 1339 sec
17 lock struct(s), heap size 3112, 28 row lock(s), undo log entries 236
MySQL thread id 28, OS thread handle 0x1a68, query id 37763097 localhost 127.0.0
.1 root cleaning up
Trx read view will not see trx with id >= 13339315174, sees < 13339315174
All other pending queries have the query strings in the status output, but this process that I believe is causing the hang, just shows this information.
If I wait a few moments and run the status again, I see the same type of transaction with some slight differences
---TRANSACTION 13339315173, ACTIVE 1572 sec
17 lock struct(s), heap size 3112, 28 row lock(s), undo log entries 265
MySQL thread id 28, OS thread handle 0x1a68, query id 38154939 localhost 127.0.0
.1 root cleaning up
Trx read view will not see trx with id >= 13339315174, sees < 13339315174
--------
I'm not very experienced with MySQL, but I'm inferring from this output that transaction 13339315173 is holding everything up. Can anyone shed some light on what may be going on?
My Rails application does not use transaction blocks, and the database itself is fairly small - less than 200 mb.
From some profiling tools, I have about 2,000 reads/sec and 30 writes/sec, with an average of 40-50 connections.
Full show innodb status below:
mysql> show engine innodb status \G
*************************** 1. row ***************************
Type: InnoDB
Name:
Status:
=====================================
2013-10-15 06:09:53 478 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 50 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 26859 srv_active, 0 srv_shutdown, 588 srv_idle
srv_master_thread log flush and writes: 27447
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 13486
OS WAIT ARRAY INFO: signal count 300223
Mutex spin waits 546801, rounds 492861, OS waits 10564
RW-shared spins 84659, rounds 148173, OS waits 2071
RW-excl spins 4010, rounds 320222, OS waits 663
Spin rounds per wait: 0.90 mutex, 1.75 RW-shared, 79.86 RW-excl
------------
TRANSACTIONS
------------
Trx id counter 13342016110
Purge done for trx's n:o < 13339316411 undo n:o < 0 state: running but idle
History list length 16262
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 71, OS thread handle 0x478, query id 38154940 localhost ::1 root
init
show engine innodb status
---TRANSACTION 13341449747, not started
MySQL thread id 68, OS thread handle 0x16d4, query id 38052625 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13341994684, not started
MySQL thread id 67, OS thread handle 0x870, query id 38131995 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13337736707, not started
MySQL thread id 66, OS thread handle 0x52c, query id 33240858 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13341868667, not started
MySQL thread id 63, OS thread handle 0x1f50, query id 37985312 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342003108, not started
MySQL thread id 62, OS thread handle 0xd64, query id 38140995 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13342015950, not started
MySQL thread id 60, OS thread handle 0x104c, query id 38154776 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342016109, not started
MySQL thread id 59, OS thread handle 0x2338, query id 38154937 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342015945, not started
MySQL thread id 58, OS thread handle 0x13f0, query id 38154767 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342015905, not started
MySQL thread id 57, OS thread handle 0x1758, query id 38154727 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13341953997, not started
MySQL thread id 56, OS thread handle 0x1e24, query id 38088539 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13341746417, not started
MySQL thread id 43, OS thread handle 0xfd8, query id 37843480 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311372513, not started
MySQL thread id 44, OS thread handle 0x9d0, query id 3625660 localhost 127.0.0.1
root cleaning up
---TRANSACTION 13341970398, not started
MySQL thread id 32, OS thread handle 0x1980, query id 38106072 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13311367975, not started
MySQL thread id 47, OS thread handle 0x2218, query id 3620225 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311372514, not started
MySQL thread id 45, OS thread handle 0x1c40, query id 3625666 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13342002623, not started
MySQL thread id 31, OS thread handle 0x192c, query id 38140455 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13311367962, not started
MySQL thread id 55, OS thread handle 0x1974, query id 3620144 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311367957, not started
MySQL thread id 54, OS thread handle 0x1aa4, query id 3620139 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311367952, not started
MySQL thread id 52, OS thread handle 0x2200, query id 3620116 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311367948, not started
MySQL thread id 51, OS thread handle 0x101c, query id 3620106 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311367998, not started
MySQL thread id 49, OS thread handle 0x23b4, query id 3620296 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311367946, not started
MySQL thread id 48, OS thread handle 0xbf8, query id 3620098 localhost 127.0.0.1
root cleaning up
---TRANSACTION 13311372515, not started
MySQL thread id 46, OS thread handle 0x2134, query id 3625651 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13341963034, not started
MySQL thread id 42, OS thread handle 0x1730, query id 38098217 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13341962985, not started
MySQL thread id 41, OS thread handle 0x2350, query id 38098156 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342004558, not started
MySQL thread id 30, OS thread handle 0x1f0c, query id 38142582 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342014064, not started
MySQL thread id 29, OS thread handle 0x1208, query id 38152727 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13342010470, not started
MySQL thread id 26, OS thread handle 0xab4, query id 38148942 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13311363353, not started
MySQL thread id 25, OS thread handle 0x1a38, query id 3614908 localhost 127.0.0.
1 root cleaning up
---TRANSACTION 13342015247, not started
MySQL thread id 24, OS thread handle 0x11ac, query id 38154020 localhost 127.0.0
.1 root cleaning up
---TRANSACTION 13341673205, ACTIVE 235 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 39, OS thread handle 0x17b8, query id 37762645 localhost 127.0.0
.1 root updating
UPDATE `stafftrack_logins` SET `end_time` = '2013-10-15 06:05:58', `updated_at`
= '2013-10-15 06:05:58' WHERE `id` = 140428
------- TRX HAS BEEN WAITING 235 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 245 page no 1355 n bits 216 index `PRIMARY` of table `rail
pro_development`.`stafftrack_logins` trx id 13341673205 lock_mode X locks rec bu
t not gap waiting
Record lock, heap no 136 PHYSICAL RECORD: n_fields 12; compact format; info bits
0
0: len 4; hex 8002248c; asc $ ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c2877; asc b (w;;
3: len 4; hex 80000011; asc ;;
4: len 4; hex 80000143; asc C;;
5: len 5; hex 9990de3e64; asc >d;;
6: len 5; hex 9990de6115; asc a ;;
7: len 1; hex 80; asc ;;
8: len 5; hex 9990de3e64; asc >d;;
9: len 5; hex 9990de6115; asc a ;;
10: SQL NULL;
11: SQL NULL;
------------------
---TRANSACTION 13341672862, ACTIVE 235 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 40, OS thread handle 0x20a8, query id 37762284 localhost 127.0.0
.1 root updating
UPDATE `rails` SET `stopped` = 1, `updated_at` = '2013-10-15 06:05:58' WHERE `id
` = 158
------- TRX HAS BEEN WAITING 235 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 225 page no 5 n bits 200 index `PRIMARY` of table `railpro
_development`.`rails` trx id 13341672862 lock_mode X locks rec but not gap waiti
ng
Record lock, heap no 95 PHYSICAL RECORD: n_fields 38; compact format; info bits
0
0: len 4; hex 8000009e; asc ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c22f6; asc b " ;;
3: len 4; hex 8000009e; asc ;;
4: len 15; hex 436c65616e20507265205374672038; asc Clean Pre Stg 8;;
5: len 1; hex 81; asc ;;
6: len 4; hex 80000005; asc ;;
7: len 4; hex 80000002; asc ;;
8: len 5; hex 998a973df8; asc = ;;
9: len 5; hex 9990de5b72; asc [r;;
10: len 4; hex 800224d9; asc $ ;;
11: len 4; hex 80000002; asc ;;
12: len 4; hex 8000005a; asc Z;;
13: len 1; hex 80; asc ;;
14: SQL NULL;
15: SQL NULL;
16: SQL NULL;
17: SQL NULL;
18: len 0; hex ; asc ;;
19: len 4; hex 43505338; asc CPS8;;
20: len 1; hex 81; asc ;;
21: len 4; hex 80000000; asc ;;
22: len 4; hex 80000000; asc ;;
23: SQL NULL;
24: len 1; hex 80; asc ;;
25: len 0; hex ; asc ;;
26: SQL NULL;
27: len 4; hex 800003e7; asc ;;
28: len 4; hex 800003e7; asc ;;
29: len 4; hex 80000002; asc ;;
30: len 4; hex 80000001; asc ;;
31: SQL NULL;
32: SQL NULL;
33: SQL NULL;
34: len 4; hex 80000005; asc ;;
35: len 1; hex 80; asc ;;
36: len 1; hex 80; asc ;;
37: len 1; hex 80; asc ;;
------------------
---TRANSACTION 13341581093, ACTIVE 300 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 69, OS thread handle 0x1f08, query id 37654138 localhost 127.0.0
.1 root updating
UPDATE `slings` SET `updated_at` = '2013-10-15 06:04:53', `tag` = 173 WHERE `id`
= 3760434
------- TRX HAS BEEN WAITING 300 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 237 page no 117 n bits 384 index `PRIMARY` of table `railp
ro_development`.`slings` trx id 13341581093 lock_mode X locks rec but not gap wa
iting
Record lock, heap no 138 PHYSICAL RECORD: n_fields 13; compact format; info bits
0
0: len 4; hex 80396132; asc 9a2;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c26e6; asc b & ;;
3: len 4; hex 800000dc; asc ;;
4: len 5; hex 9990de59d8; asc Y ;;
5: len 5; hex 9990de6073; asc `s;;
6: SQL NULL;
7: len 4; hex 80000009; asc ;;
8: len 4; hex 80000017; asc ;;
9: len 4; hex 800000b7; asc ;;
10: len 5; hex 9990de6073; asc `s;;
11: len 1; hex 30; asc 0;;
12: SQL NULL;
------------------
---TRANSACTION 13341529477, ACTIVE 332 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 38, OS thread handle 0x1fb0, query id 37598517 localhost 127.0.0
.1 root updating
UPDATE `stafftrack_logins` SET `end_time` = '2013-10-15 06:04:21', `updated_at`
= '2013-10-15 06:04:21' WHERE `id` = 140428
------- TRX HAS BEEN WAITING 332 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 245 page no 1355 n bits 208 index `PRIMARY` of table `rail
pro_development`.`stafftrack_logins` trx id 13341529477 lock_mode X locks rec bu
t not gap waiting
Record lock, heap no 136 PHYSICAL RECORD: n_fields 12; compact format; info bits
0
0: len 4; hex 8002248c; asc $ ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c2877; asc b (w;;
3: len 4; hex 80000011; asc ;;
4: len 4; hex 80000143; asc C;;
5: len 5; hex 9990de3e64; asc >d;;
6: len 5; hex 9990de6115; asc a ;;
7: len 1; hex 80; asc ;;
8: len 5; hex 9990de3e64; asc >d;;
9: len 5; hex 9990de6115; asc a ;;
10: SQL NULL;
11: SQL NULL;
------------------
---TRANSACTION 13341500243, ACTIVE 349 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 37, OS thread handle 0x1624, query id 37565181 localhost 127.0.0
.1 root updating
UPDATE `stafftrack_logins` SET `end_time` = '2013-10-15 06:04:04', `updated_at`
= '2013-10-15 06:04:04' WHERE `id` = 140421
------- TRX HAS BEEN WAITING 349 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 245 page no 1355 n bits 200 index `PRIMARY` of table `rail
pro_development`.`stafftrack_logins` trx id 13341500243 lock_mode X locks rec bu
t not gap waiting
Record lock, heap no 134 PHYSICAL RECORD: n_fields 12; compact format; info bits
0
0: len 4; hex 80022485; asc $ ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c2823; asc b (#;;
3: len 4; hex 8000003e; asc >;;
4: len 4; hex 800000d3; asc ;;
5: len 5; hex 9990de37d9; asc 7 ;;
6: len 5; hex 9990de6103; asc a ;;
7: len 1; hex 80; asc ;;
8: len 5; hex 9990de37d9; asc 7 ;;
9: len 5; hex 9990de6103; asc a ;;
10: SQL NULL;
11: SQL NULL;
------------------
---TRANSACTION 13341499101, ACTIVE 350 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 36, OS thread handle 0x19a8, query id 37563945 localhost 127.0.0
.1 root updating
UPDATE `stafftrack_logins` SET `end_time` = '2013-10-15 06:04:03', `updated_at`
= '2013-10-15 06:04:03' WHERE `id` = 140421
------- TRX HAS BEEN WAITING 350 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 245 page no 1355 n bits 200 index `PRIMARY` of table `rail
pro_development`.`stafftrack_logins` trx id 13341499101 lock_mode X locks rec bu
t not gap waiting
Record lock, heap no 134 PHYSICAL RECORD: n_fields 12; compact format; info bits
0
0: len 4; hex 80022485; asc $ ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c2823; asc b (#;;
3: len 4; hex 8000003e; asc >;;
4: len 4; hex 800000d3; asc ;;
5: len 5; hex 9990de37d9; asc 7 ;;
6: len 5; hex 9990de6103; asc a ;;
7: len 1; hex 80; asc ;;
8: len 5; hex 9990de37d9; asc 7 ;;
9: len 5; hex 9990de6103; asc a ;;
10: SQL NULL;
11: SQL NULL;
------------------
---TRANSACTION 13341486252, ACTIVE 360 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 35, OS thread handle 0x1538, query id 37550281 localhost 127.0.0
.1 root updating
UPDATE `stafftrack_logins` SET `end_time` = '2013-10-15 06:03:53', `updated_at`
= '2013-10-15 06:03:53' WHERE `id` = 140433
------- TRX HAS BEEN WAITING 360 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 245 page no 1355 n bits 200 index `PRIMARY` of table `rail
pro_development`.`stafftrack_logins` trx id 13341486252 lock_mode X locks rec bu
t not gap waiting
Record lock, heap no 133 PHYSICAL RECORD: n_fields 12; compact format; info bits
0
0: len 4; hex 80022491; asc $ ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c27f9; asc b ' ;;
3: len 4; hex 8000003e; asc >;;
4: len 4; hex 80000079; asc y;;
5: len 5; hex 9990de3eb3; asc > ;;
6: len 5; hex 9990de60ed; asc ` ;;
7: len 1; hex 80; asc ;;
8: len 5; hex 9990de3eb3; asc > ;;
9: len 5; hex 9990de60ed; asc ` ;;
10: SQL NULL;
11: SQL NULL;
------------------
---TRANSACTION 13341476145, ACTIVE 368 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 27, OS thread handle 0x1794, query id 37539510 localhost 127.0.0
.1 root updating
UPDATE `stafftrack_logins` SET `end_time` = '2013-10-15 06:03:45', `updated_at`
= '2013-10-15 06:03:45' WHERE `id` = 140433
------- TRX HAS BEEN WAITING 368 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 245 page no 1355 n bits 200 index `PRIMARY` of table `rail
pro_development`.`stafftrack_logins` trx id 13341476145 lock_mode X locks rec bu
t not gap waiting
Record lock, heap no 133 PHYSICAL RECORD: n_fields 12; compact format; info bits
0
0: len 4; hex 80022491; asc $ ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c27f9; asc b ' ;;
3: len 4; hex 8000003e; asc >;;
4: len 4; hex 80000079; asc y;;
5: len 5; hex 9990de3eb3; asc > ;;
6: len 5; hex 9990de60ed; asc ` ;;
7: len 1; hex 80; asc ;;
8: len 5; hex 9990de3eb3; asc > ;;
9: len 5; hex 9990de60ed; asc ` ;;
10: SQL NULL;
11: SQL NULL;
------------------
---TRANSACTION 13341210786, ACTIVE 561 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 61, OS thread handle 0xcc0, query id 37228004 localhost 127.0.0.
1 root updating
UPDATE `slings` SET `tag` = 221, `updated_at` = '2013-10-15 06:00:32' WHERE `id`
= 3760256
------- TRX HAS BEEN WAITING 561 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 237 page no 113 n bits 408 index `PRIMARY` of table `railp
ro_development`.`slings` trx id 13341210786 lock_mode X locks rec but not gap wa
iting
Record lock, heap no 240 PHYSICAL RECORD: n_fields 13; compact format; info bits
0
0: len 4; hex 80396080; asc 9` ;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c2365; asc b #e;;
3: len 4; hex 800000dc; asc ;;
4: len 5; hex 9990de5644; asc VD;;
5: len 5; hex 9990de5c11; asc \ ;;
6: SQL NULL;
7: len 4; hex 80000008; asc ;;
8: len 4; hex 8000001f; asc ;;
9: len 4; hex 800000de; asc ;;
10: len 5; hex 9990de5c11; asc \ ;;
11: len 1; hex 30; asc 0;;
12: SQL NULL;
------------------
---TRANSACTION 13340733149, ACTIVE 813 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 34, OS thread handle 0x1c74, query id 36686947 localhost 127.0.0
.1 root updating
UPDATE `slings` SET `tag` = 0, `updated_at` = '2013-10-15 05:56:20' WHERE `id` =
3750500
------- TRX HAS BEEN WAITING 813 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 237 page no 103 n bits 368 index `PRIMARY` of table `railp
ro_development`.`slings` trx id 13340733149 lock_mode X locks rec but not gap wa
iting
Record lock, heap no 299 PHYSICAL RECORD: n_fields 13; compact format; info bits
0
0: len 4; hex 80393a64; asc 9:d;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c258d; asc b % ;;
3: len 4; hex 800000dc; asc ;;
4: len 5; hex 9990d6ad21; asc !;;
5: len 5; hex 9990de5e09; asc ^ ;;
6: SQL NULL;
7: len 4; hex 80000007; asc ;;
8: len 4; hex 8000001e; asc ;;
9: len 4; hex 800000d3; asc ;;
10: len 5; hex 9990de5e09; asc ^ ;;
11: len 1; hex 30; asc 0;;
12: SQL NULL;
------------------
---TRANSACTION 13340568617, ACTIVE 903 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 33, OS thread handle 0xfd4, query id 36497676 localhost 127.0.0.
1 root updating
UPDATE `slings` SET `tag` = 183, `updated_at` = '2013-10-15 05:54:50' WHERE `id`
= 3760434
------- TRX HAS BEEN WAITING 903 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 237 page no 117 n bits 328 index `PRIMARY` of table `railp
ro_development`.`slings` trx id 13340568617 lock_mode X locks rec but not gap wa
iting
Record lock, heap no 138 PHYSICAL RECORD: n_fields 13; compact format; info bits
0
0: len 4; hex 80396132; asc 9a2;;
1: len 6; hex 00031b15cbe5; asc ;;
2: len 7; hex 620000078c26e6; asc b & ;;
3: len 4; hex 800000dc; asc ;;
4: len 5; hex 9990de59d8; asc Y ;;
5: len 5; hex 9990de6073; asc `s;;
6: SQL NULL;
7: len 4; hex 80000009; asc ;;
8: len 4; hex 80000017; asc ;;
9: len 4; hex 800000b7; asc ;;
10: len 5; hex 9990de6073; asc `s;;
11: len 1; hex 30; asc 0;;
12: SQL NULL;
------------------
---TRANSACTION 13339315173, ACTIVE 1572 sec
17 lock struct(s), heap size 3112, 28 row lock(s), undo log entries 265
MySQL thread id 28, OS thread handle 0x1a68, query id 38154939 localhost 127.0.0
.1 root cleaning up
Trx read view will not see trx with id >= 13339315174, sees < 13339315174
--------
FILE I/O
--------
I/O thread 0 state: wait Windows aio (insert buffer thread)
I/O thread 1 state: wait Windows aio (log thread)
I/O thread 2 state: wait Windows aio (read thread)
I/O thread 3 state: wait Windows aio (read thread)
I/O thread 4 state: wait Windows aio (read thread)
I/O thread 5 state: wait Windows aio (read thread)
I/O thread 6 state: wait Windows aio (write thread)
I/O thread 7 state: wait Windows aio (write thread)
I/O thread 8 state: wait Windows aio (write thread)
I/O thread 9 state: wait Windows aio (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
16009 OS file reads, 247119 OS file writes, 196868 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 4.14 writes/s, 4.14 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 600, seg size 602, 461 merges
merged operations:
insert 234, delete mark 72731, delete 6105
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 25499819, node heap has 1457 buffer(s)
18360.79 hash searches/s, 6378.53 non-hash searches/s
---
LOG
---
Log sequence number 32913782367
Log flushed up to 32913782357
Pages flushed up to 32905803711
Last checkpoint at 32905803711
0 pending log writes, 0 pending chkp writes
154012 log i/o's done, 4.14 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 13186891776; in additional pool allocated 0
Dictionary memory allocated 704979
Buffer pool size 786432
Free buffers 768193
Database pages 16782
Old database pages 6349
Modified db pages 1585
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 177, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 15892, created 890, written 74469
0.00 reads/s, 0.02 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 16782, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 98304
Free buffers 95915
Database pages 2211
Old database pages 836
Modified db pages 191
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 34, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2119, created 92, written 12239
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2211, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 98304
Free buffers 96184
Database pages 1931
Old database pages 732
Modified db pages 234
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 15, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1831, created 100, written 5972
0.00 reads/s, 0.02 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 1931, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 98304
Free buffers 96026
Database pages 2098
Old database pages 794
Modified db pages 104
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 28, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2090, created 8, written 2450
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2098, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size 98304
Free buffers 95810
Database pages 2311
Old database pages 872
Modified db pages 354
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 11, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2012, created 299, written 20722
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2311, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size 98304
Free buffers 96204
Database pages 1915
Old database pages 726
Modified db pages 128
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 24, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1850, created 65, written 10904
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 1915, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size 98304
Free buffers 95833
Database pages 2288
Old database pages 864
Modified db pages 196
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 20, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2201, created 87, written 6023
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2288, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size 98304
Free buffers 95952
Database pages 2173
Old database pages 821
Modified db pages 224
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 23, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2079, created 94, written 10846
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2173, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size 98304
Free buffers 96269
Database pages 1855
Old database pages 704
Modified db pages 154
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 22, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1710, created 145, written 5313
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 1855, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
1 read views open inside InnoDB
Main thread id 7604, state: sleeping
Number of rows inserted 12203, updated 144283, deleted 45540, read 8849139354
0.36 inserts/s, 4.46 updates/s, 0.04 deletes/s, 119285.59 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
1 row in set (0.00 sec)
mysql>
In case anyone comes across this in the future. We found the problem was related to mutli-threading and a bug in Rails 2.3.2 handling connections.
The jist of it was that a connection was being released before we were actually done using it, so sometimes in the middle of a transaction the mysql thread was being passed to a different process, and getting itself in to nested transaction cascading locks throughout the application.
To fix this we updated to rails 2.3.18, and backported a lot of Rails 3 ActiveRecord.

MySQL updates getting stuck

I have a Mediacore-based website that is getting stuck due to an update query. Those update queries are running all the time, but sometimes (I believe, during high traffic hours), one of those updates just becomes stuck, locking all other queries:
245 mediacore_user localhost mediacore Query 36 Locked UPDATE media SET modified_on='2011-12-21 09:42:58', views=(media.views + 1) WHERE media.id = 1048
246 mediacore_user localhost mediacore Query 36 Locked UPDATE media SET modified_on='2011-12-21 09:42:58', views=(media.views + 1) WHERE media.id = 1048
248 mediacore_user localhost mediacore Query 36 Updating UPDATE media SET modified_on='2011-12-21 09:42:58', views=(media.views + 1) WHERE media.id = 1048
249 mediacore_user localhost mediacore Query 36 Locked UPDATE media SET modified_on='2011-12-21 09:42:58', views=(media.views + 1) WHERE media.id = 1048
253 mediacore_user localhost mediacore Query 36 Locked UPDATE media SET modified_on='2011-12-21 09:42:58', views=(media.views + 1) WHERE media.id = 1048
258 mediacore_user localhost mediacore Query 30 Locked UPDATE media SET modified_on='2011-12-21 09:43:04', views=(media.views + 1) WHERE media.id = 416
268 mediacore_user localhost mediacore Query 3 Locked UPDATE media SET modified_on='2011-12-21 09:43:31', views=(media.views + 1) WHERE media.id = 1048
269 mediacore_user localhost mediacore Query 3 Locked UPDATE media SET modified_on='2011-12-21 09:43:31', views=(media.views + 1) WHERE media.id = 1048
270 mediacore_user localhost mediacore Query 3 Locked UPDATE media SET modified_on='2011-12-21 09:43:31', views=(media.views + 1) WHERE media.id = 1048
271 mediacore_user localhost mediacore Query 3 Locked UPDATE media SET modified_on='2011-12-21 09:43:31', views=(media.views + 1) WHERE media.id = 1048
272 mediacore_user localhost mediacore Query 1 Locked UPDATE media SET modified_on='2011-12-21 09:43:33', views=(media.views + 1) WHERE media.id = 1048
If I kill the "Updating" connection, another one gets stuck, and so on, until there are only about 3 queries left locked.
Any ideas?
Media table:
CREATE TABLE `media` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`type` varchar(8) DEFAULT NULL,
`slug` varchar(50) NOT NULL,
`podcast_id` int(11) DEFAULT NULL,
`reviewed` tinyint(1) NOT NULL,
`encoded` tinyint(1) NOT NULL,
`publishable` tinyint(1) NOT NULL,
`created_on` datetime NOT NULL,
`modified_on` datetime NOT NULL,
`publish_on` datetime DEFAULT NULL,
`publish_until` datetime DEFAULT NULL,
`title` varchar(255) NOT NULL,
`subtitle` varchar(255) DEFAULT NULL,
`description` text,
`description_plain` text,
`notes` text,
`duration` int(11) NOT NULL,
`views` int(11) NOT NULL,
`likes` int(11) NOT NULL,
`popularity_points` int(11) NOT NULL,
`author_name` varchar(50) NOT NULL,
`author_email` varchar(255) NOT NULL,
`dislikes` int(11) NOT NULL,
`popularity_likes` int(11) NOT NULL,
`popularity_dislikes` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `slug` (`slug`),
KEY `podcast_id` (`podcast_id`),
CONSTRAINT `media_ibfk_1` FOREIGN KEY (`podcast_id`) REFERENCES `podcasts` (`id`) ON DELETE SET NULL ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=1049 DEFAULT CHARSET=utf8
INNODB Status after the problem happened:
=====================================
111221 12:12:09 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 6 seconds
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 78, signal count 78
Mutex spin waits 0, rounds 453, OS waits 17
RW-shared spins 122, OS waits 61; RW-excl spins 0, OS waits 0
------------
TRANSACTIONS
------------
Trx id counter 0 1115749
Purge done for trx's n:o < 0 1115735 undo n:o < 0 0
History list length 13
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0 1115747, not started, process no 15360, OS thread id 140486827099904
MySQL thread id 111, query id 11647 localhost mediacore_user
---TRANSACTION 0 0, not started, process no 15360, OS thread id 140486827898624
MySQL thread id 106, query id 11651 localhost root
show innodb status
---TRANSACTION 0 1115709, not started, process no 15360, OS thread id 140486826567424
MySQL thread id 103, query id 10874 localhost mediacore_user
---TRANSACTION 0 1115731, not started, process no 15360, OS thread id 140486792349440
MySQL thread id 102, query id 11517 localhost mediacore_user
---TRANSACTION 0 1115668, not started, process no 15360, OS thread id 140486792881920
MySQL thread id 101, query id 10067 localhost 127.0.0.1 a7user
---TRANSACTION 0 1115737, not started, process no 15360, OS thread id 140486791816960
MySQL thread id 95, query id 11578 localhost mediacore_user
---TRANSACTION 0 1115738, not started, process no 15360, OS thread id 140486827366144
MySQL thread id 87, query id 11577 localhost mediacore_user
---TRANSACTION 0 1115744, not started, process no 15360, OS thread id 140486827632384
MySQL thread id 61, query id 11606 localhost mediacore_user
---TRANSACTION 0 1115746, not started, process no 15360, OS thread id 140486826301184
MySQL thread id 22, query id 11633 localhost mediacore_user
---TRANSACTION 0 1115748, ACTIVE 1 sec, process no 15360, OS thread id 140486826833664
mysql tables in use 1, locked 1
MySQL thread id 55, query id 11650 localhost mediacore_user Table lock
UPDATE media SET modified_on='2011-12-21 12:12:08', dislikes=10 WHERE media.id = 572
Trx read view will not see trx with id >= 0 1115749, sees < 0 1115740
---TRANSACTION 0 1115743, ACTIVE 14 sec, process no 15360, OS thread id 140486826034944
mysql tables in use 1, locked 1
MySQL thread id 23, query id 11601 localhost mediacore_user Table lock
UPDATE media SET modified_on='2011-12-21 12:11:55', views=(media.views + 1) WHERE media.id = 1048
Trx read view will not see trx with id >= 0 1115744, sees < 0 1115740
---TRANSACTION 0 1115742, ACTIVE 14 sec, process no 15360, OS thread id 140486825502464
mysql tables in use 1, locked 1
MySQL thread id 90, query id 11599 localhost mediacore_user Table lock
UPDATE media SET modified_on='2011-12-21 12:11:55', views=(media.views + 1) WHERE media.id = 1048
Trx read view will not see trx with id >= 0 1115743, sees < 0 1115740
---TRANSACTION 0 1115741, ACTIVE 14 sec, process no 15360, OS thread id 140486825768704 starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 368, 1 row lock(s)
MySQL thread id 73, query id 11596 localhost mediacore_user Updating
UPDATE media SET modified_on='2011-12-21 12:11:55', views=(media.views + 1) WHERE media.id = 1048
Trx read view will not see trx with id >= 0 1115742, sees < 0 1115740
------- TRX HAS BEEN WAITING 14 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 1025 n bits 88 index `PRIMARY` of table `mediacore`.`media` trx id 0 1115741 lock_mode X locks rec but not gap waiting
Record lock, heap no 20 PHYSICAL RECORD: n_fields 27; compact format; info bits 0
0: len 4; hex 80000418; asc ;; 1: len 6; hex 00000011065c; asc \;; 2: len 7; hex 00000003c20d5b; asc [;; 3: len 5; hex 766964656f; asc video;; 4: len 30; hex 617273656c696f2d7061746f2d64652d63617276616c686f2d686f6d656e; asc arselio-pato-de-carvalho-homen;...(truncated); 5: len 4; hex 80000013; asc ;; 6: len 1; hex 81; asc ;; 7: len 1; hex 81; asc ;; 8: len 1; hex 81; asc ;; 9: len 8; hex 8000124a82217b0b; asc J !{ ;; 10: len 8; hex 8000124a8230e083; asc J 0 ;; 11: len 8; hex 8000124a82220ef0; asc J " ;; 12: SQL NULL; 13: len 30; hex 417273c3a96c696f205061746f2064652043617276616c686f20686f6d65; asc Ars lio Pato de Carvalho home;...(truncated); 14: SQL NULL; 15: len 30; hex 3c703e43656e74726f206465204e6575726f6369c3aa6e63696173206520; asc <p>Centro de Neuroci ncias e ;...(truncated); 16: len 30; hex 43656e74726f206465204e6575726f6369c3aa6e6369617320652042696f; asc Centro de Neuroci ncias e Bio;...(truncated); 17: SQL NULL; 18: len 4; hex 80000000; asc ;; 19: len 4; hex 80000099; asc ;; 20: len 4; hex 80000000; asc ;; 21: len 4; hex 80000000; asc ;; 22: len 3; hex 554356; asc UCV;; 23: len 9; hex 7563764075632e7074; asc ucv#uc.pt;; 24: len 4; hex 80000000; asc ;; 25: len 4; hex 80000000; asc ;; 26: len 4; hex 80000000; asc ;;
------------------
---TRANSACTION 0 1115740, ACTIVE 14 sec, process no 15360, OS thread id 140486792083200
2 lock struct(s), heap size 368, 1 row lock(s), undo log entries 1
MySQL thread id 97, query id 11597 localhost mediacore_user Table lock
SELECT 1
FROM media_fulltext
LIMIT 1
Trx read view will not see trx with id >= 0 1115741, sees < 0 1115741
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (write thread)
Pending normal aio reads: 0, aio writes: 0,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
187 OS file reads, 469 OS file writes, 298 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2,
0 inserts, 0 merged recs, 0 merges
Hash table size 2212699, node heap has 9 buffer(s)
0.17 hash searches/s, 0.17 non-hash searches/s
---
LOG
---
Log sequence number 0 148655397
Log flushed up to 0 148655397
Last checkpoint at 0 148655397
0 pending log writes, 0 pending chkp writes
177 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 1225697720; in additional pool allocated 11846912
Dictionary memory allocated 203840
Buffer pool size 65536
Free buffers 65299
Database pages 228
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages read 225, created 3, written 259
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
6 read views open inside InnoDB
Main thread process no. 15360, id 140486801540864, state: sleeping
Number of rows inserted 13, updated 85, deleted 2, read 281971
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.17 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
EDIT: Somehow, some of the updates are not unlocking the table after they commit. I watched the queries for quite some time and saw that after 50 seconds, the lock is freed and one update is executed. I've changed the innodb_lock_wait_timeout to 5 and I saw the same thing, only with intervals of 5 seconds instead of 50. But once it gets only a few updates left locked, they're all executed instantaneously... it's a very strange behavior...
A bit late but I'd like to add some more information: From my observation this happens if you have the full-text index enabled. In that case there seems to be a bad interaction in MySQL involving InnoDB (media table) and MyISAM (full-text index) locking.
I never saw the issue if MediaCore's full-text triggers were not installed.
I think it is because of the large number of queries at a time. Try using mysql index with your fields. It will speed up the performance.