mysql deadlock with foreign key - mysql

DDL(removed some irrelevant columns info)
Table: config_version
Create Table: CREATE TABLE `config_version` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`config_id` bigint(20) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `fk_config_version_config_1` (`config_id`),
CONSTRAINT `fk_config_version_config_1` FOREIGN KEY (`config_id`) REFERENCES `config` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=85532 DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT
Table: config
Create Table: CREATE TABLE `config` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`),
UNIQUE KEY `unique_config_name_namespace_id` (`name`,`namespace_id`),
KEY `fk_config_namespace_1` (`namespace_id`),
CONSTRAINT `fk_config_namespace_1` FOREIGN KEY (`namespace_id`) REFERENCES `namespace` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1901 DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT
show engine innodb status;
*** (1) TRANSACTION:
TRANSACTION 125411, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 5 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1
MySQL thread id 7, OS thread handle 123145570861056, query id 4898 localhost 127.0.0.1 root updating
update config set name = 'bts_driver_list_filter_gray_onceagain_list_filter_tw_control_group', description = '?????????????', namespace_id = 4, current_version = 89950, create_time = '2017-11-14 15:54:45', created_by = 'chenchongyu' where id = 982
2017-12-01T04:12:43.711301Z 6 [Note] InnoDB: *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 672 page no 11 n bits 216 index PRIMARY of table `apollo_config_dev`.`config` trx id 125411 lock_mode X locks rec but not gap waiting
Record lock, heap no 97 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 8; hex 80000000000003d6; asc ;;
1: len 6; hex 00000001e9da; asc ;;
2: len 7; hex 24000001830584; asc $ ;;
3: len 30; hex 6274735f6472697665725f6c6973745f66696c7465725f677261795f6f6e; asc bts_driver_list_filter_gray_on; (total 66 bytes);
4: len 13; hex 3f3f3f3f3f3f3f3f3f3f3f3f3f; asc ?????????????;;
5: len 11; hex 6368656e63686f6e677975; asc chenchongyu;;
6: len 8; hex 8000000000000004; asc ;;
7: len 8; hex 8000000000015f56; asc _V;;
8: len 5; hex 999e1cfdad; asc ;;
9: len 1; hex 01; asc ;;
2017-12-01T04:12:43.712251Z 6 [Note] InnoDB: *** (2) TRANSACTION:
TRANSACTION 125408, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
5 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1
MySQL thread id 6, OS thread handle 123145570582528, query id 4904 localhost 127.0.0.1 root updating
update config set name = 'bts_driver_list_filter_gray_onceagain_list_filter_tw_control_group', description = '?????????????', namespace_id = 4, current_version = 89947, create_time = '2017-11-14 15:54:45', created_by = 'chenchongyu' where id = 982
2017-12-01T04:12:43.712308Z 6 [Note] InnoDB: *** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 672 page no 11 n bits 216 index PRIMARY of table `apollo_config_dev`.`config` trx id 125408 lock mode S locks rec but not gap
Record lock, heap no 97 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 8; hex 80000000000003d6; asc ;;
1: len 6; hex 00000001e9da; asc ;;
2: len 7; hex 24000001830584; asc $ ;;
3: len 30; hex 6274735f6472697665725f6c6973745f66696c7465725f677261795f6f6e; asc bts_driver_list_filter_gray_on; (total 66 bytes);
4: len 13; hex 3f3f3f3f3f3f3f3f3f3f3f3f3f; asc ?????????????;;
5: len 11; hex 6368656e63686f6e677975; asc chenchongyu;;
6: len 8; hex 8000000000000004; asc ;;
7: len 8; hex 8000000000015f56; asc _V;;
8: len 5; hex 999e1cfdad; asc ;;
9: len 1; hex 01; asc ;;
2017-12-01T04:12:43.713122Z 6 [Note] InnoDB: *** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 672 page no 11 n bits 216 index PRIMARY of table `apollo_config_dev`.`config` trx id 125408 lock_mode X locks rec but not gap waiting
Record lock, heap no 97 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 8; hex 80000000000003d6; asc ;;
1: len 6; hex 00000001e9da; asc ;;
2: len 7; hex 24000001830584; asc $ ;;
3: len 30; hex 6274735f6472697665725f6c6973745f66696c7465725f677261795f6f6e; asc bts_driver_list_filter_gray_on; (total 66 bytes);
4: len 13; hex 3f3f3f3f3f3f3f3f3f3f3f3f3f; asc ?????????????;;
5: len 11; hex 6368656e63686f6e677975; asc chenchongyu;;
6: len 8; hex 8000000000000004; asc ;;
7: len 8; hex 8000000000015f56; asc _V;;
8: len 5; hex 999e1cfdad; asc ;;
9: len 1; hex 01; asc ;;
2017-12-01T04:12:43.714133Z 6 [Note] InnoDB: *** WE ROLL BACK TRANSACTION (2)
简化业务流程代码(code)
#Transaction
public void service(){
Config config = configMapper.selectByPrimaryKey(configId)
ConfigVersion configVersion = toConfigVersion(config, configDTO)
configVersionMapper.insert(configVersion)
config.setCurrentVersion(configVersion.id)
config.setDescription(configDTO.description)
configMapper.updateByPrimaryKey(config)
wait for service C http response
}
sql of the code
insert into config_version (config_id, name, create_time, namespace_id, type, status,base_version_id, value, created_by, enable_from, description)
values
( #{configId,jdbcType=BIGINT},
#{name,jdbcType=VARCHAR}, now(),
#{namespaceId,jdbcType=BIGINT}, #{type,jdbcType=TINYINT},
#{status,jdbcType=TINYINT},
#{baseVersionId,jdbcType=BIGINT}, #{value,jdbcType=LONGVARCHAR}, #{createdBy,jdbcType=VARCHAR},
#{enableFrom,jdbcType=TIMESTAMP},#{description,jdbcType=VARCHAR})
update config set
name = #{name,jdbcType=VARCHAR},
description = #{description,jdbcType=VARCHAR},
namespace_id = #{namespaceId,jdbcType=BIGINT},
current_version = #{currentVersion,jdbcType=BIGINT},
create_time = #{createTime,jdbcType=TIMESTAMP},
created_by = #{createdBy,jdbcType=VARCHAR}
where id = #{id,jdbcType=BIGINT}
业务场景(service invoke order)
服务A 调用服务B 更新 config version,服务B 将更新内容持久化到mysql 以后会调用服务C,等待服务C返回以后,服务B再返回给服务A
如果服务A 一定时间内没有接受到服务B返回,那么服务A 会重试,重新向服务A 发起一个请求
(service A -> A, service B -> B, service C ->C;
A invoke B, to update config_version,B persist to mysql, and then invoke C,after C returned, B return to A.
A will retry if B did not return;
)
解决办法(solution)
去掉config_version 对config的外键依赖(remove the foreign key)
CONSTRAINT `fk_config_version_config_1` FOREIGN KEY (`config_id`) REFERENCES `config` (`id`)
Question
但是没明白mysql insert 和update 时需要获取什么锁,所以没明白这里产生死锁的原因。
(the reason why deadlock happened, which kind of lock get in a tx)
软件版本信息(mysql info)
sql
mysql 版本信息
mysql: stable 5.7.20

Related

Unexpected database locking in symfony

I have a recurring database lock that I'm trying to understand.
I have a cron that is adding jobs to a queue. These jobs are then processed by various workers on different instances of production. All of them are importing data from a resource, and then adding/updating a table. Theoretically, there would never be two jobs writing to the same row, as the data is partitioned in such a way that this won't happen (jobs are created per provider, which means the id used in this table is unique).
This is what msyql is logging for the lock:
2023-01-12T10:49:41.876818Z 610974 [Note] InnoDB:
*** (1) TRANSACTION:
TRANSACTION 41621347924, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1
MySQL thread id 610972, OS thread handle 22890357896960, query id 17518666 {instance} {app} updating
UPDATE offer SET timestamp = '2023-01-12 10:49:41' WHERE id = 17479128
2023-01-12T10:49:41.876850Z 610974 [Note] InnoDB: *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 3037 page no 11514 n bits 96 index PRIMARY of table `{app}_production`.`offer` trx id 41621347924 lock_mode X locks rec but not gap waiting
Record lock, heap no 16 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 810ab5d8; asc ;;
1: len 6; hex 0009b0d35e55; asc ^U;;
2: len 7; hex 230002401d04c1; asc # # ;;
3: len 4; hex 8000e54d; asc M;;
4: len 17; hex 3330313032393431313035333030323630; asc 30102941105300260;;
5: len 3; hex 8fce3c; asc <;;
6: len 3; hex 8fce41; asc A;;
7: len 30; hex 3c7265736f75726365207265736f7572636569643d223330313032393431; asc <resource resourceid="30102941; (total 326 bytes);
8: len 1; hex 81; asc ;;
9: len 5; hex 99af18ac69; asc i;;
2023-01-12T10:49:41.877237Z 610974 [Note] InnoDB: *** (2) TRANSACTION:
TRANSACTION 41621347925, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1
MySQL thread id 610974, OS thread handle 22894496118528, query id 17518667 {instance} {app} updating
UPDATE offer SET timestamp = '2023-01-12 10:49:41' WHERE id = 17821328
2023-01-12T10:49:41.877262Z 610974 [Note] InnoDB: *** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 3037 page no 11514 n bits 96 index PRIMARY of table `{app}_production`.`offer` trx id 41621347925 lock_mode X locks rec but not gap
Record lock, heap no 16 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 810ab5d8; asc ;;
1: len 6; hex 0009b0d35e55; asc ^U;;
2: len 7; hex 230002401d04c1; asc # # ;;
3: len 4; hex 8000e54d; asc M;;
4: len 17; hex 3330313032393431313035333030323630; asc 30102941105300260;;
5: len 3; hex 8fce3c; asc <;;
6: len 3; hex 8fce41; asc A;;
7: len 30; hex 3c7265736f75726365207265736f7572636569643d223330313032393431; asc <resource resourceid="30102941; (total 326 bytes);
8: len 1; hex 81; asc ;;
9: len 5; hex 99af18ac69; asc i;;
2023-01-12T10:49:41.877643Z 610974 [Note] InnoDB: *** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 3037 page no 10300 n bits 104 index PRIMARY of table `{app}_production`.`offer` trx id 41621347925 lock_mode X locks rec but not gap waiting
Record lock, heap no 4 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 810fee90; asc ;;
1: len 6; hex 0009b0d35e54; asc ^T;;
2: len 7; hex 22000240070d5b; asc " # [;;
3: len 4; hex 8000e54d; asc M;;
4: len 17; hex 3330313032393431313035333030323630; asc 30102941105300260;;
5: len 3; hex 8fce38; asc 8;;
6: len 3; hex 8fce3c; asc <;;
7: len 30; hex 3c7265736f75726365207265736f7572636569643d223330313032393431; asc <resource resourceid="30102941; (total 326 bytes);
8: len 1; hex 81; asc ;;
9: len 5; hex 99af18ac69; asc i;;
2023-01-12T10:49:41.878014Z 610974 [Note] InnoDB: *** WE ROLL BACK TRANSACTION (2)
I have always sucked at sorting out deadlocking issues so looking for a bit of guidance.
Is this a table or a row lock? I'm using symfony 2.8 (don't ask) and it's doctrine packages.
Is there some way to disable the locking completely?
What can I read to better understand these logs?

INSERT ON DUPLICATE KEY UPDATE occurs a deadlock error, which is about record lock

I use insert ... on duplicate key updateto insert data in batches.
But I get a error when do it with more than one threads, actually just two threads do it.
The table is:
create table alarm {
id varchar(20) not null collate utf8_unicode_ci,
`date` date not null,
adas blob,
dsm blob,
bsd blob,
primary key(id, `date`)
) engine=innodb default charset=utf8
collate=utf8_unicode_ci
The insert statement is like:
insert into alarm(id, `date`, adas) values
('10001', 20220105, 0x1231231),
('10002', 20220105, 0xAFED001),
....
on duplicate key update
adas=concat(ifnull(adas,''), ifnull(values(adas),''));
The another insert statement is like:
insert into alarm(id, `date`, dsm) values
('10001', 20220105, 0x1231231),
('10002', 20220105, 0xAFED001),
('10008', 20220105, 0xAFED001),
('00008', 20220105, 0xAFED34),
....
on duplicate key update
dsm=concat(ifnull(dsm,''), ifnull(values(dsm),''));
sometimes, it will occur a dead lock error. The database log as following:
> myql show engine innodb status \G;
------------------------
LATEST DETECTED DEADLOCK
------------------------
2022-02-22 16:49:29 0x1fac
*** (1) TRANSACTION:
TRANSACTION 3902837, ACTIVE 1 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 23 lock struct(s), heap size 1080, 28 row lock(s), undo log entries 31
MySQL thread id 85930, OS thread handle 17464, query id 34130124 DESKTOP-AJ1S2TF 192.168.20.191 root update
INSERT INTO device_alarm (did, `date`, dsm) VALUES('4044383', '20220222', 0x3a000a043078313012043078303220422945b8c9a832303f4031832f4ca60a685e40389ac7d2900642120a0e33303334333133313336333233301001),('4050503', '20220222', 0x38000a043078313012043078303229342db1321a493f4031ca4e3fa88b685e4038a1c7d2900642120a0e33303334333133323331333633371001),('2000023', '20220222', 0x3a000a0430783034120430783031202f29e08618af79fd36403159ddea39e9695c4038a5c7d2900642120a0e33303332333333313333333433351005),('4026821', '20220222', 0x38000a04307831301204307830322982aca7565f293f40316347e350bf725e40388cc7d2900642120a0e34313432353934413337333833331001),('4051611', '20220222', 0x3a000a0430783130120430783032201229377007ea94473f403113d38558fd545e403891c7d2900642120a0e33303334333133313339333633351001),('6493492', '20220222', 0x38000a043078303412043078303220432939d4efc2d65e4040317e54c37e4f5e5c40388ec7d29006421
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 682 page no 31 n bits 112 index PRIMARY of table `httpserver`.`device_alarm` /* `p20220223` */ trx id 3902837 lock_mode X locks rec but not gap waiting
Record lock, heap no 42 PHYSICAL RECORD: n_fields 7; compact format; info bits 0
0: len 11; hex 3430373336373431363336; asc 4073674;;
1: len 3; hex 8fcc56; asc V;;
2: len 6; hex 0000003b8d76; asc ; v;;
3: len 7; hex 5500000bba09b8; asc U ;;
4: len 30; hex 46000a04307830321204307830312a0430783031320430783030403149d1; asc F 0x02 0x01* 0x012 0x00#1I ; (total 576 bytes);
5: len 30; hex 38000a043078313012043078303229fe7bf0daa5e13e403124d1cb289672; asc 8 0x10 0x02) { >#1$ ( r; (total 412 bytes);
6: SQL NULL;
*** (2) TRANSACTION:
TRANSACTION 3902838, ACTIVE 1 sec inserting
mysql tables in use 1, locked 1
36 lock struct(s), heap size 3296, 39 row lock(s), undo log entries 48
MySQL thread id 85931, OS thread handle 8108, query id 34130125 DESKTOP-AJ1S2TF 192.168.20.191 root update
INSERT INTO device_alarm (did, `date`, adas) VALUES('1810534', '20220222', 0x46000a04307830321204307830322a0430783032320430783030404349a2410a9e42aa424051a2410a9e42aa42405895c7d2900662120a0e33353333333433303338333633381005),('2000041', '20220222', 0x46000a04307830321204307830312a04307830323204307830304020491920d1048a343f40511920d1048a343f40589ac7d2900662120a0e33303334333133313334333333391005),('1800684', '20220222', 0x48000a0430783032120430783031200b2a0430783031320430783030402e498bfcfa2136b24240518bfcfa2136b2424058a5c7d2900662120a0e33363338333433393336333133381005),('1030014', '20220222', 0x48000a043078303112043078303120062a0430783030320430783030402c4945b8c9a832ac38405145b8c9a832ac38405892c7d2900662120a0e33303331333433303333333433311005),('2000024', '20220222', 0x46000a04307830321204307830312a0430783032320430783030401f49284696ccb100374051284696ccb10037405897c7d2900662120a0e33303332333
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 682 page no 31 n bits 112 index PRIMARY of table `httpserver`.`device_alarm` /* `p20220223` */ trx id 3902838 lock_mode X locks rec but not gap
Record lock, heap no 42 PHYSICAL RECORD: n_fields 7; compact format; info bits 0
0: len 11; hex 3430373336373431363336; asc 4073674;;
1: len 3; hex 8fcc56; asc V;;
2: len 6; hex 0000003b8d76; asc ; v;;
3: len 7; hex 5500000bba09b8; asc U ;;
4: len 30; hex 46000a04307830321204307830312a0430783031320430783030403149d1; asc F 0x02 0x01* 0x012 0x00#1I ; (total 576 bytes);
5: len 30; hex 38000a043078313012043078303229fe7bf0daa5e13e403124d1cb289672; asc 8 0x10 0x02) { >#1$ ( r; (total 412 bytes);
6: SQL NULL;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 682 page no 136 n bits 120 index PRIMARY of table `httpserver`.`device_alarm` /* `p20220223` */ trx id 3902838 lock_mode X locks rec but not gap waiting
Record lock, heap no 50 PHYSICAL RECORD: n_fields 7; compact format; info bits 0
0: len 11; hex 3138373030383530303738; asc 1870085;;
1: len 3; hex 8fcc56; asc V;;
2: len 6; hex 0000003b8d75; asc ; u;;
3: len 7; hex 540000223f2ba1; asc T "?+ ;;
4: len 30; hex 46000a04307830321204307830312a0430783032320430783030402f4983; asc F 0x02 0x01* 0x022 0x00#/I ; (total 360 bytes);
5: len 30; hex 3a000a04307830341204307830312027292ae109bdfe0e4340317a8b87f7; asc : 0x04 0x01 ')* C#1z ; (total 602 bytes);
6: SQL NULL;
*** WE ROLL BACK TRANSACTION (1)
The lock is lock_mode X locks rec but not gap, it's too strange for me to undertand it. Why did the dead lock happen?

is there any way to show MySQL table lock history

I am using MySQL 5.7.26,is there any way to show table lock history?I know how to check current table lock using:
show OPEN TABLES where In_use > 0;
I want to avoid table lock in my SQL query,so view the history may help.Current lock may not get all locked table(Maybe the locked table query not execute right now).
Today I found a way to show this in MySQL 5.7:
show engine innodb status
this is the part of result output:
====================================
2020-09-25 10:36:53 0x7ff4b4c5d700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 12 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 762213 srv_active, 0 srv_shutdown, 139997 srv_idle
srv_master_thread log flush and writes: 902210
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 363879865
OS WAIT ARRAY INFO: signal count 448819672
RW-shared spins 0, rounds 425739262, OS waits 209673117
RW-excl spins 0, rounds 4271714439, OS waits 133598673
RW-sx spins 135926, rounds 2757000, OS waits 52788
Spin rounds per wait: 425739262.00 RW-shared, 4271714439.00 RW-excl, 20.28 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2020-09-25 10:10:41 0x7ff4b4ce1700
*** (1) TRANSACTION:
TRANSACTION 841893940, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s)
MySQL thread id 108348, OS thread handle 140688980530944, query id 184994716 172.19.104.233 root Searching rows for update
UPDATE r_room_seat SET status = 1,
id = (SELECT #update_id := id),
user_id = 62659,
robot_flag = 1
WHERE room_play_id = 539 and status = 0 and online = 1
LIMIT 1
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 3493 page no 528 n bits 176 index PRIMARY of table `room_db`.`r_room_seat` trx id 841893940 lock_mode X locks rec but not gap waiting
Record lock, heap no 56 PHYSICAL RECORD: n_fields 20; compact format; info bits 0
0: len 8; hex 000000000000d649; asc I;;
1: len 6; hex 0000322d9b9a; asc 2- ;;
2: len 7; hex 3e000040042eda; asc > # . ;;
3: len 8; hex 800000000000021b; asc ;;
4: len 8; hex 8000000000000097; asc ;;
5: len 8; hex 800000000000f4d5; asc ;;
6: len 8; hex 7fffffffffffffff; asc ;;
7: len 8; hex 8000017452aa2c5e; asc tR ,^;;
8: len 4; hex 80000000; asc ;;
9: len 4; hex 80000000; asc ;;
10: len 8; hex 8000017452aa2c5e; asc tR ,^;;
11: len 8; hex 8000017452aa2c5e; asc tR ,^;;
12: len 4; hex 80000001; asc ;;
13: len 1; hex 81; asc ;;
14: len 8; hex 8000000000000005; asc ;;
15: len 10; hex 48415050595f5241494e; asc HAPPY_RAIN;;
16: len 4; hex 80000001; asc ;;
17: len 8; hex 8000000000000000; asc ;;
18: len 8; hex 8000000000000074; asc t;;
19: len 4; hex 80000001; asc ;;
*** (2) TRANSACTION:
TRANSACTION 841893905, ACTIVE 0 sec fetching rows
mysql tables in use 2, locked 2
746 lock struct(s), heap size 90320, 128960 row lock(s)
MySQL thread id 100540, OS thread handle 140688982152960, query id 184994612 172.19.104.233 root Sending data
UPDATE r_room_seat s
SET s.status = 1,
s.room_play_id = (SELECT #update_id := max(room_play_id)),
s.user_id = (
case when s.seat_num = 1
then 63182
when s.seat_num = 2
then 16472
else -1 end
),
s.robot_flag = (
case when s.seat_num = 1
then 1
when s.seat_num = 2
then 0
else 0 end
)
WHERE s.status = 0
and s.online = 1
and s.room_play_id in (
select room_play_id
from (
select room_play_id as room_play_id
from r_room_seat
where room_id = 166
and status = 0
group by room_play_id
having count(*) = 2
limit 1
) a
)
and s.room_id = 166
LIMIT 2
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 3493 page no 528 n bits 176 index PRIMARY of table `room_db`.`r_room_seat` trx id 841893905 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
you can get the dead lock information and analysis.

MySQL deadlocking issue with InnoDB

I have a central database server and several "worker" servers which are executing queries like this concurrently:
UPDATE job_queue
SET
worker = '108.166.81.112',
attempts = attempts + 1,
started = '2014-01-14 10:34:03',
token = '13eb3e6a8c3e1becb34051e08f19fd62'
WHERE completed = '0000-00-00 00:00:00'
AND (started = '0000-00-00 00:00:00' OR started < '2014-01-14 10:29:03')
AND attempts < 2
ORDER BY priority DESC, inserted
LIMIT 1
Occasionally my job_queue table locks up and if I run "SHOW ENGINE INNODB STATUS", I get something like this:
------------------------
LATEST DETECTED DEADLOCK
------------------------
140114 10:34:15
*** (1) TRANSACTION:
TRANSACTION 0 46984514, ACTIVE 0 sec, process no 590, OS thread id 140366633146112 fetching rows
mysql tables in use 1, locked 1
LOCK WAIT 20 lock struct(s), heap size 3024, 545 row lock(s)
MySQL thread id 677401, query id 19385205 10.179.103.110 root init
UPDATE job_queue SET worker='108.166.81.112', attempts=attempts+1, started='2014-01-14 10:34:03', token='13eb3e6a8c3e1becb34051e08f19fd62' WHERE completed='0000-00-00 00:00:00' AND (started='0000-00-00 00:00:00' OR started<'2014-01-14 10:29:03') AND attempts<2 ORDER BY priority DESC, inserted LIMIT 1
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 245767 n bits 128 index `PRIMARY` of table `database`.`job_queue` trx id 0 46984514 lock_mode X waiting
Record lock, heap no 34 PHYSICAL RECORD: n_fields 12; compact format; info bits 0
0: len 3; hex 800210; asc ;; 1: len 6; hex 000002cced25; asc %;; 2: len 7; hex 000003c00f1970; asc p;; 3: len 30; hex 4f3a31343a2243425343616368654170704a6f62223a363a7b733a31393a; asc O:14:"CBSCacheAppJob":6:{s:19:;...(truncated); 4: len 1; hex 80; asc ;; 5: len 8; hex 800012513c58bf24; asc Q<X $;; 6: len 8; hex 800012513c58cc17; asc Q<X ;; 7: len 14; hex 31302e3137392e3130332e313333; asc 10.179.103.133;; 8: len 1; hex 81; asc ;; 9: len 8; hex 800012513c58cc32; asc Q<X 2;; 10: len 0; hex ; asc ;; 11: len 30; hex 353264393033616162656634346239626536306463346438666432303066; asc 52d903aabef44b9be60dc4d8fd200f;...(truncated);
*** (2) TRANSACTION:
TRANSACTION 0 46984485, ACTIVE 17 sec, process no 590, OS thread id 140366633547520 starting index read, thread declared inside InnoDB 500
mysql tables in use 1, locked 1
4 lock struct(s), heap size 1216, 2 row lock(s), undo log entries 1
MySQL thread id 676723, query id 19385209 10.179.103.133 root init
UPDATE job_queue SET worker='10.179.103.133', attempts=attempts+1, started='2014-01-14 10:34:03', token='efd21d0d34f44badbc30386db4dd252e' WHERE completed='0000-00-00 00:00:00' AND (started='0000-00-00 00:00:00' OR started<'2014-01-14 10:29:03') AND attempts<2 ORDER BY priority DESC, inserted LIMIT 1
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 245767 n bits 128 index `PRIMARY` of table `database`.`job_queue` trx id 0 46984485 lock_mode X locks rec but not gap
Record lock, heap no 34 PHYSICAL RECORD: n_fields 12; compact format; info bits 0
0: len 3; hex 800210; asc ;; 1: len 6; hex 000002cced25; asc %;; 2: len 7; hex 000003c00f1970; asc p;; 3: len 30; hex 4f3a31343a2243425343616368654170704a6f62223a363a7b733a31393a; asc O:14:"CBSCacheAppJob":6:{s:19:;...(truncated); 4: len 1; hex 80; asc ;; 5: len 8; hex 800012513c58bf24; asc Q<X $;; 6: len 8; hex 800012513c58cc17; asc Q<X ;; 7: len 14; hex 31302e3137392e3130332e313333; asc 10.179.103.133;; 8: len 1; hex 81; asc ;; 9: len 8; hex 800012513c58cc32; asc Q<X 2;; 10: len 0; hex ; asc ;; 11: len 30; hex 353264393033616162656634346239626536306463346438666432303066; asc 52d903aabef44b9be60dc4d8fd200f;...(truncated);
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 57 n bits 120 index `PRIMARY` of table `database`.`job_queue` trx id 0 46984485 lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 12; compact format; info bits 0
0: len 3; hex 800001; asc ;; 1: len 6; hex 000002ccdab1; asc ;; 2: len 7; hex 000003c0352b3f; asc 5+?;; 3: len 30; hex 4f3a31323a224175746f50696c6f744a6f62223a363a7b733a31383a2200; asc O:12:"AutoPilotJob":6:{s:18:" ;...(truncated); 4: len 1; hex 82; asc ;; 5: len 8; hex 800012513c58af57; asc Q<X W;; 6: len 8; hex 800012513c58bf22; asc Q<X ";; 7: len 14; hex 3130382e3136362e38312e313132; asc 108.166.81.112;; 8: len 1; hex 81; asc ;; 9: len 8; hex 800012513c58bf23; asc Q<X #;; 10: len 0; hex ; asc ;; 11: len 30; hex 616331376430346339326163613366323330646164323239363764336266; asc ac17d04c92aca3f230dad22967d3bf;...(truncated);
*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------
Trx id counter 0 46989905
Purge done for trx's n:o < 0 46986227 undo n:o < 0 0
History list length 24
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0 0, not started, process no 590, OS thread id 140366628529920
MySQL thread id 703864, query id 20047015 localhost root
SHOW ENGINE INNODB STATUS
---TRANSACTION 0 46989894, not started, process no 590, OS thread id 140366636758784
MySQL thread id 702822, query id 20046897 10.179.1.63 root
---TRANSACTION 0 46986223, ACTIVE 39782 sec, process no 590, OS thread id 140366626322176
25 lock struct(s), heap size 3024, 710 row lock(s), undo log entries 9
MySQL thread id 677706, query id 19994505 10.179.103.114 root
Trx read view will not see trx with id >= 0 46986224, sees < 0 46986224
Any further writes to the table then time out on me until I restart my MySQL server (or manually kill the deadlocked jobs):
PHP Fatal error: Lock wait timeout exceeded; try restarting transaction(Query: "UPDATE job_queue SET worker='108.166.81.250', attempts=attempts+1, started='2014-01-14 21:27:45', token='369eae55a7f0eacad3b678a3410de8e4' WHERE completed='0000-00-00 00:00:00' AND (started='0000-00-00 00:00:00' OR started<'2014-01-14 21:22:45') AND attempts<2 ORDER BY priority DESC, inserted LIMIT 1") in /utilities/Database.php on line 53
Can anyone explain to me why this query causes a deadlock? I was under the impression that all queries on an InnoDB table happened atomically. Any ideas?
This is causing a deadlock because the UPDATE query is locking all rows in the table, and depending on the indexes used (or lack thereof), two different sessions will potentially lock them in slightly different order. Remember that UPDATE, DELETE, and SELECT ... FOR UPDATE will lock all rows they encounter, whether those rows match all WHERE conditions or not. So when using them, you should try hard to be sure that they encounter as few rows as possible, by using indexes (ideally the primary key) and avoiding vague or wide-selecting conditions.
My suggestion for work queues is pretty much universal: Lock as little as possible as rarely as possible and always in a deterministic order. So, generally:
Use non-locking reads (regular SELECT) to find work to do by looking for stuff your worker knows how to do and is currently unclaimed (lease_owner IS NULL AND lease_expiry IS NULL -- or similar).
Choose one work item (or a few if you dare, but one is far simpler and normally allows for perfectly acceptable performance).
Update your work item (to claim it, but for any case it needs updating as well):
Open a transaction.
Lock your chosen work item with SELECT ... FOR UPDATE -- If it is no longer unclaimed, abort and choose another.
Update your chosen work item with your worker id and an expiry time for your lease on it.
Commit your transaction immediately.
Begin work on your leased work items.
In some other process, another poller looks for abandoned work and un-claims it (via the same update process above).
You can easily get very high throughput with this design (thousands of jobs per second), and with essentially no contention and no ordering issues. Optimizations to choose work that is less likely to conflict with other pollers are simple and effective (e.g. modulus on Job ID or similar, chosen to avoid starvation of jobs). The key is to remember that conflict on job selection is okay -- just abort and try again and everything moves along very quickly.
All locking writes for work queue items/jobs should be done only on single rows and by primary key only.

Mysql transaction waiting for lock which is already granted .. This is causing deadlock

If following situation a bug in mysql?.
Mysql Version: mysql.x86_64 5.0.77-4.el5_4.1
Kernel: Linux box2 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
------------------------
LATEST DETECTED DEADLOCK
------------------------
100125 4:24:41
*** (1) TRANSACTION:
TRANSACTION 0 210510625, ACTIVE 155 sec, process no 28125, OS thread id 1243162944 starting index read
mysql tables in use 1, locked 1
LOCK WAIT 5 lock struct(s), heap size 1216, undo log entries 1
MySQL thread id 162928579, query id 527252744 box22 172.16.11.105 user updating
delete from user_grid_items where user_id = 669786974 and START_X = 45 and START_Y = 65
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 61372 n bits 328 index `PRIMARY` of table `gamesutra_beta/user_grid_items` trx id 0 210510625 lock_mode X locks rec but not gap waiting
Record lock, heap no 127 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 8; hex 0000000027ec235e; asc ' #^;; 1: len 4; hex 0000002d; asc -;; 2: len 4; hex 00000041; asc A;; 3: len 6; hex 00000b561243; asc V C;; 4: len 7; hex 80000040070110; asc # ;; 5: len 23; hex 474949445f414e494d414c535f53515549445f50494e4b; asc GIID_ANIMALS_SQUID_PINK;; 6: len 4; hex cb59f060; asc Y `;; 7: len 4; hex 4b59f060; asc KY `;; 8: len 4; hex 80000000; asc ;; 9: len 1; hex 80; asc ;;
*** (2) TRANSACTION:
TRANSACTION 0 210505911, ACTIVE 555 sec, process no 28125, OS thread id 1184323904 starting index read, thread declared inside InnoDB 500
mysql tables in use 1, locked 1
5 lock struct(s), heap size 1216, undo log entries 1
MySQL thread id 162924258, query id 527252762 box22 172.16.11.105 user updating
delete from user_grid_items where user_id = 669786974 and START_X = 45 and START_Y = 65
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 61372 n bits 328 index `PRIMARY` of table `gamesutra_beta/user_grid_items` trx id 0 210505911 lock mode S locks rec but not gap
Record lock, heap no 127 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 8; hex 0000000027ec235e; asc ' #^;; 1: len 4; hex 0000002d; asc -;; 2: len 4; hex 00000041; asc A;; 3: len 6; hex 00000b561243; asc V C;; 4: len 7; hex 80000040070110; asc # ;; 5: len 23; hex 474949445f414e494d414c535f53515549445f50494e4b; asc GIID_ANIMALS_SQUID_PINK;; 6: len 4; hex cb59f060; asc Y `;; 7: len 4; hex 4b59f060; asc KY `;; 8: len 4; hex 80000000; asc ;; 9: len 1; hex 80; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 61372 n bits 328 index `PRIMARY` of table `gamesutra_beta/user_grid_items` trx id 0 210505911 lock_mode X locks rec but not gap waiting
Record lock, heap no 127 PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 8; hex 0000000027ec235e; asc ' #^;; 1: len 4; hex 0000002d; asc -;; 2: len 4; hex 00000041; asc A;; 3: len 6; hex 00000b561243; asc V C;; 4: len 7; hex 80000040070110; asc # ;; 5: len 23; hex 474949445f414e494d414c535f53515549445f50494e4b; asc GIID_ANIMALS_SQUID_PINK;; 6: len 4; hex cb59f060; asc Y `;; 7: len 4; hex 4b59f060; asc KY `;; 8: len 4; hex 80000000; asc ;; 9: len 1; hex 80; asc ;;
*** WE ROLL BACK TRANSACTION (2)
------------
Sometimes the SHOW ENGINE INNODB STATUS can be hard to decipher because it only shows the current statement in the transaction. It does not show statements that occurred previously in the same transaction that may have acquired the locks that are actually being held.
In your case, a previous statement in transaction 2 acquired a shared lock on the row in question.
Then, transaction 1 attempted to acquire an exclusive lock on the same row, and is happily waiting for the shared lock to be removed.
Then, transaction 2, in another statement, attempted to acquire an exclusive lock on the same row. Classic deadlock. Neither transaction can finish.
One solution to help avoid such a deadlock would be to grab an exclusive lock on the row in transaction 2 with a SELECT FOR UPDATE statement, prior to the statement that's acquiring the shared lock.
I read something long ago, and not sure if it MIGHT be what you are running into as a result or not... without seeing code of the transaction of the current vs what it is conflicting with.
When processing your transactions, you should try to have them always do any locking in the same sequence... For an order / order detail / payments system, do the activities in the order listed as example here for all similar. If you have another process that tries its transaction in order of "Order detail / order", that CAN cause a deadlock.
One transaction is locking the order # first, then working the order detail. The other transaction locking the order detail first, then trying to get the order header.
HTH
Use SHOW ENGINE INNODB STATUS to determine the cause of the latest deadlock. That can help you to tune your application to avoid deadlocks.
Always be prepared to re-issue a transaction if it fails due to deadlock. Deadlocks are not dangerous. Just try again.