InnoDB mutual table lock from two clients - mysql

Question

I'm stuck with the following problem and hope, that someone help me to solve it...
Mysql version 5.6.19-0ubuntu0.14.04.1-log.
CREATE TABLE `dr` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`start_date` bigint(20) unsigned NOT NULL,
`end_date` bigint(20) unsigned NOT NULL,
UNIQUE KEY `id` (`id`,`start_date`),
KEY `start_date` (`start_date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED;
CREATE TABLE `map_dr2pdu` (
`drtable_id` int(10) unsigned NOT NULL,
`dr_id` bigint(20) unsigned NOT NULL,
`packet_location` bigint(20) unsigned NOT NULL,
`sctp_chunk` smallint(5) unsigned DEFAULT NULL,
KEY `dr_id` (`dr_id`),
KEY `drtable_id` (`drtable_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED;
CREATE TABLE `dr_hot` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`start_date` bigint(20) unsigned NOT NULL,
`end_date` bigint(20) unsigned NOT NULL,
UNIQUE KEY `id` (`id`,`start_date`)
) ENGINE=MEMORY;
CREATE TABLE `map_dr2pdu_hot` (
`drtable_id` int(10) unsigned NOT NULL,
`dr_id` bigint(20) unsigned NOT NULL,
`packet_location` bigint(20) unsigned NOT NULL,
`sctp_chunk` smallint(5) unsigned DEFAULT NULL,
KEY `dr_id` (`dr_id`),
KEY `drtable_id` (`drtable_id`)
) ENGINE=MEMORY;
dr and map_dr2pdu have partitions (if it matters).
dr_hot.id <===> map_dr2pdu_hot.dr_id
There is a client C1 (auto commit off)
writes into dr_hot
obtains LAST_INSERTED_ID and inserts 1-n records into map_dr2pdu_hot.
C1 commit politics: 500 inserts or 1 sec (which came first)
Memory tables are introduced solely to support high insertion rate. Actual data must be store in dr and map_dr2pdu (INNODB) tables. To make this "rotation"
there is second client C2 (auto commit off) which
locks _hot table,
stores its autoincrement id
dumps its data into file in memory,
truncates _hot table (faster, than delete)
updates its autoincrement id
commit
unlocks the table
[loads data from the file to INNODB table]
Problem appears when C2 begins its work:
mysql> show processlist;
+--------+-----------------+-----------------+-----------+---------+-------+----------------+---------------+----+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+-----------------+-----------+---------+-------+---------------------------------+----+
| 1 | event_scheduler | localhost | NULL | Daemon | 1 | Waiting for next activation | NULL |
| 57 | riva | localhost | riva | Sleep | 283 | | NULL |
| 115748 | root | localhost | riva | Query | 0 | init | show processlist |
| 117538 | riva | localhost | riva | Sleep | 10889 | | NULL |
| 120150 | riva | localhost | riva | Sleep | 30 | | NULL |
| 120155 | riva | localhost | riva | Sleep | 7 | | NULL |
| 120158 | riva | localhost | riva | Sleep | 5 | | NULL |
| 120161 | riva | localhost | riva | Sleep | 5 | | NULL |
| 120164 | riva | localhost | riva | Sleep | 5 | | NULL |
| 120167 | riva | localhost | riva | Sleep | 7 | | NULL |
| 120170 | riva | localhost | riva | Query | 22 | Waiting for table metadata lock | ALTER TABLE dr_hot AUTO_INCREMENT=1152885790 |
| 120178 | riva | localhost | riva | Sleep | 5 | | NULL |
| 120179 | riva | localhost | riva | Query | 23 | Waiting for table metadata lock | LOCK TABLES map_dr2pdu_hot LOW_PRIORITY WRITE |
| 120243 | riva | localhost | riva | Sleep | 24 | | NULL |
| 120244 | riva | localhost:40934 | riva | Sleep | 24 | | NULL |
| 120245 | riva | localhost:40935 | riva | Sleep | 24 | | NULL |
| 120246 | riva | localhost:40936 | riva | Sleep | 22 | | NULL |
| 120247 | riva | localhost:40937 | riva | Sleep | 24 | | NULL |
| 120248 | riva | localhost:40938 | riva | Sleep | 24 | | NULL |
| 120249 | riva | localhost:40939 | riva | Sleep | 24 | | NULL |
| 120250 | riva | localhost:40940 | riva | Sleep | 24 | | NULL |
| 120251 | riva | localhost:40941 | riva | Sleep | 22 | | NULL |
| 120252 | riva | localhost:40942 | riva | Sleep | 22 | | NULL |
| 120253 | riva | localhost:40943 | riva | Execute | 22 | Waiting for table metadata lock | INSERT INTO map_dr2pdu_hot (xdrtable_id, xdr_id, packet_location) VALUES (25, 6078, |
+--------+-----------------+-----------------+-----------+---------+-------+---------------------------------+------+
24 rows in set (0.00 sec)
mysql general_log + processlist:
119177 Execute INSERT INTO map_dr2pdu_hot (drtable_id, dr_id, packet_location) VALUES (26, 131287232, 606
4773029186237937)
119177 Query COMMIT
119453 Query LOCK TABLES map_dr2pdu_hot WRITE
processlist:
| 119453 | rivasense | localhost | rivasense | Query | 164 | Waiting for table metadata lock | LOCK TABLES map_dr2pdu_hot WRITE
119177 Query START TRANSACTION
119177 Execute INSERT INTO dr_hot ()
Show engine innodb status, does not show any deadlock, just like
mysql> select * from INNODB_LOCKS;
Empty set (0.04 sec)
mysql> select * from INNODB_LOCK_WAITS;
Empty set (0.00 sec)
As I understoo : C1 acquires lock for insert and seems not commits, but it DOES according to log. therefore, C2 can't lock table.
Please help to find the root cause.
update on 06/10/14:
Basically problem reproducible when C1 tries to write something into map_dr2pdu_hot and C2 simultaneously tries to lock the same table. This treated as "almost" deadlock (not listed among deadlocks) and released with lock_wait_timeout expired.
Could you suggest how to resolve such a problem?

Answer 1

Finally we had to apply WA - modify algorithm of C2 to:
stores _hot tbl autoincrement id
dumps its data into file in memory,
deletes from _hot table (where id less than auto inc id from step 1)
loads data from the file to INNODB table
of course map_dr2pdu table altered to have id auto-increment column.