Increase in deadlocks when adding primary key. Why? - mysql

Question

First, a bit of necessary background (please, bear with me). I work as a developers of a web application using MySQL for persistance. We have implemented audit logging by creating an audit trail table for each data table. We might for example have the following table definitions for a Customer entity:
-- Data table definition.
CREATE TABLE my_database.customers (
CustomerId INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
FirstName VARCHAR(255) NOT NULL,
LastName VARCHAR(255) NOT NULL,
-- More data columns, removed for simplicity.
...
);
-- Audit table definition in separate schema.
CREATE TABLE my_database_audittrail.customers (
CustomerId INT(11) DEFAULT NULL,
FirstName VARCHAR(255) DEFAULT NULL,
LastName VARCHAR(255) DEFAULT NULL,
-- More data columns.
...
-- Audit meta data columns.
ChangeTime DATETIME NOT NULL,
ChangeByUser VARCHAR(255) NOT NULL
);
As you can see, the audit table is simply a copy of the data table plus some metadata. Note that the audit table doesn't have any keys. When, for example, we update a customer, our ORM generates SQL similar to the following:
-- Insert a copy of the customer entity, before the update, into the audit table.
INSERT INTO my_database_audittrail.customers (
CustomerId,
FirstName,
LastName,
...
ChangeTime,
ChangeByUser)
)
SELECT
CustomerId,
FirstName,
LastName,
...
NOW(),
#ChangeByUser
FROM my_database.customers
WHERE CustomerId = #CustomerId;
-- Then update the data table.
UPDATE
my_database.customers
SET
FirstName = #FirstName,
LastName = #LastName,
...
WHERE CustomerId = #CustomerId;
This has worked well enough. Recently, however, we needed to add a primary key column to the audit tables for various reasons, changing the audit table definition to something similar to the following:
CREATE TABLE my_database_audittrail.customers (
__auditId INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
CustomerId INT(11) DEFAULT NULL,
FirstName VARCHAR(255) DEFAULT NULL,
LastName VARCHAR(255) DEFAULT NULL,
...
ChangeTime DATETIME NOT NULL,
ChangeByUser VARCHAR(255) NOT NULL
);
The SQL generated by our ORM when updating data tables has not been modified. This change seem to have increased the risk of deadlock very much. The system in question is a web application with a number of nightly batch jobs. The increase in deadlocks doesn't show in the day to day use of the system by our web users. The nightly batch jobs, however, do suffer from the deadlocks very much as they do intense work on a few database tables. Our "solution" has been to add a retry-upon-deadlock strategy (hardly controversial) and while this seems to work fine I would very much like to understand why the above change has increased the risk of deadlocks that much (and if we can somehow remedy the problem).
Further information:
Our nightly batch jobs do INSERTS, UPDATES and DELETES on our data tables. Only INSERTS are performed on the audit tables.
We use repeatable read isolation level on out database transactions.
Before this change, we haven't seen a single deadlock when running our nightly batch jobs.
UPDATE: Checked SHOW ENGINE INNODB STATUS to determine the cause of the deadlocks and found this:
*** WAITING FOR THIS LOCK TO BE GRANTED:
TABLE LOCK table `my_database_audittrail`.`customers` trx id 24972756464 lock mode AUTO-INC waiting
I was under the impression that auto increments was handled outside of any transactions in order to avoid using the same auto increment value in different transactions? But I guess the AUTO_INCREMENT property on the primary key we introduced seems to be the problem?

Answer 1

This is speculation.
Inserting or updating into a table with indexes not only locks the data pages but also the index pages, including the higher levels of the index. When multiple threads are affecting records at the same time, they may lock different portions of the index.
This would not generally show up with single record inserts. However, two statements that are updating multiple records might start acquiring locks on the index and find that they are deadlocking each other. Retry may be sufficient for fixing this problem. Alternatively, it seems that "too much" may be running at one time and you may want to consider how the nightly update work is laid out.

Answer 2

When inserting into tables with auto increment columns, MySQL uses different strategies to acquire values for the auto increments column(s) depending on which type of insert is made, on your insert statements and how MySQL is configured to handle auto increment columns, an insert may result in a complete table lock.
With "simple inserts", i.e inserts where MySQL can determine before hand the number of rows which will be inserted into a table (e.g INSERT INTO table (col1, col2) VALUES (val1, val2);) auto increment column values are acquired using a light weight lock on the auto increment counter. This light weight lock is released as soon as the auto increment values are acquired so one won't have to wait until the actual insert to complete. I.e no table lock.
However, with "bulk inserts", where MySQL cannot determine the number of inserted rows before hand (e.g INSERT INTO table (col1, col2) SELECT col1, col2 FROM table2 WHERE ...;) a table lock is created to acquire auto increment column values and not relinquished until the insert is completed.
The above is per MySQL's default configuration. MySQL can be configured to not use table locks on bulk inserts but this may cause auto increment columns to have different values on masters and slaves (if replication is set up) and thus may or may not be an viable option.