I've run into an innodb locking issue for transactions on a table with both a primary key and a separate unique index. It seems if a TX deletes a record using a unique key, and then re-inserts that same record, this will result in a next-key lock instead of the expected record lock (since the key is unique). See below for a test case as well as breakdown of what records I expect to have what locks:
DROP TABLE IF EXISTS foo;
CREATE TABLE `foo` (
`i` INT(11) NOT NULL,
`j` INT(11) DEFAULT NULL,
PRIMARY KEY (`i`),
UNIQUE KEY `jk` (`j`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 ;
INSERT INTO foo VALUES (5,5), (8,8), (11,11);
(Note: Just run the TX2 sql after the TX1 sql, in a separate connection)
TX1
START TRANSACTION;
DELETE FROM foo WHERE i=8;
results in exclusive lock on i=8 (no gap lock since i is primary key and unique)
INSERT INTO foo VALUES(8,8);
results in exclusive lock for i=8 & j= 8, and shared intention lock on i=6 & i=7, as well as j=6 & j=7
TX2
START TRANSACTION;
INSERT INTO foo VALUES(7,7);
results in exclusive lock for i=7 & j=7, as well as shared intention lock on on i=6 & j=6
I would expect TX2 to not be blocked by TX1, however it is. Oddly, the blocking seems to be related to the insert by TX1. I say this because if TX1's insert statement is not run after the delete, TX2's insert is not blocked. It's almost as if TX1's re-insertion of (8,8) causes a next-key lock on index j for (6,8].
Any insight would be much appreciated.
The problem you are experiencing happens because MySQL doesn't just lock the table row for a value you're going to insert, it locks all possible values between the previous id and the next id in order, so, reusing your example bellow:
DROP TABLE IF EXISTS foo;
CREATE TABLE `foo` (
`i` INT(11) NOT NULL,
`j` INT(11) DEFAULT NULL,
PRIMARY KEY (`i`),
UNIQUE KEY `jk` (`j`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 ;
INSERT INTO foo VALUES (5,5), (8,8), (11,11);
Suppose you start with transaction TX1:
START TRANSACTION;
REPLACE INTO foo VALUES(8,8);
Then if you start a transaction TX2, whatever INSERT or REPLACE using an id between 5 and 11 will be locked:
START TRANSACTION;
REPLACE INTO foo VALUES(11,11);
Looks like MySQL uses this kind of locking to avoid the "phantom problem" described here: http://dev.mysql.com/doc/refman/5.0/en/innodb-next-key-locking.html, MySQL uses a "next-key locking", that combines index-row locking with gap locking, this means for us that it will lock a lot of possible ids between the previous and next ids, and will lock prev and next ids as well.
To avoid this try to create a server algorithm that inserts your records so that records inserted in different transactions don't overlap, or at least don't execute all your transactions at the same time so the TX doesn't have to wait one each other.
It seems as if the problem might lie in the fact that InnoDB indexes are weird.
The primary key (clustered) is i and there would be a rowid associated with it.
The unique key on j (nonclustered) has the rowid of i associated with the value of j in the index.
Doing a DELETE followed by an INSERT on the same key value for i should produce an upcoming different rowid for the primary key (clustered) and, likewise, an upcoming different rowid to associate with the value of j (nonclustered).
This would require some bizarre internal locking within MVCC mechanism.
You may need to change your Transaction Isolation Level to Allow Dirty Reads (i.e., not have repeatable reads)
Play some games with tx_isolation variable within a session
Try READ_COMMITTED and READ_UNCOMMITTED
Click here to see syntax for setting Isolation Level in a Session
Click here to see how there was once a bug concerning this within a Session and the warning on how to use it carefully
Otherwise, just permamnently set the following in /etc/my.cnf (Example)
[mysqld]
transaction_isolation=read-committed
Give it a try !!!
https://bugs.mysql.com/bug.php?id=68021
this bug issue answer your question.
This is the design flaw of InnoDB, the upstream used to fixed this issue to avoid gap lock in row_ins_scan_sec_index_for_duplicate in read-committed isolation. However it bring out another issue, the fix cause secondary index unique key violation silently, so the upstream revert this fix..
Related
In mysql document https://dev.mysql.com/doc/refman/8.0/en/innodb-locks-set.html. It says
SELECT ... FOR UPDATE and SELECT ... FOR SHARE statements that use a
unique index acquire locks for scanned rows, and release the locks for
rows that do not qualify for inclusion in the result set (for example,
if they do not meet the criteria given in the WHERE clause).
But in practice, it does not work as I expected. Here is my test.
Create a new Table Person.
CREATE TABLE `Persion` (
`id` bigint NOT NULL,
`name` varchar(100) NOT NULL,
`age` bigint NOT NULL,
PRIMARY KEY (`id`),
KEY `Persion_name_IDX` (`name`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
Insert following data.
id
name
age
10
bob
19
11
cat
20
This is my sql statement.
//transaction 1
start transaction;
SELECT * from test.Persion p where id IN (10,11) and age=19 for UPDATE;
//transaction 2
UPDATE test.Persion set name='tx6' where id=11;
In transaction 1, when I execute the select...for update sql, it should not lock the id=11 record, because it does't match the age=19 condition. So my second transaction should not be blocked. But in fact, it is!
So I want to know what happend. Did I understand the document correct?
I read the documentation the same way you do, and I think it's unclear or incorrect. I repeated your test and I got the same result you did — the row with id 11 remained locked by the first session.
Then I tested your case after setting the following in the first session:
SET transaction_isolation='READ-COMMITTED';
This allows the second session to update the row.
This makes me think locking follows the same rule for updates in READ-COMMITTED isolation level, described on this page: https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html
READ COMMITTED
Using READ COMMITTED has additional effects:
For UPDATE or DELETE statements, InnoDB holds locks only for rows that it updates or deletes. Record locks for nonmatching rows are released after MySQL has evaluated the WHERE condition.
This a follow up to a previous question of mine...
I have a table with to fields: pos (point of sale) and voucher_number. I need to have an unique sequence per pos.
CREATE TABLE `test_table` (
`id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`date_event` DATETIME NOT NULL,
`pos` TINYINT NOT NULL,
`voucher_number` INT NOT NULL,
`origin` CHAR(1) NULL,
PRIMARY KEY (`id`))
ENGINE = InnoDB;
I perform the tests described in my answer and everything works fine. Basically there are two or more scripts trying to do the following at the same time:
//Notice the SELECT ... FOR UPDATE
$max_voucher_number = select max(voucher_number) as max_voucher_number from vouchers where pos = $pos for update;
$max_voucher_number = $max_voucher_number + 1;
insert into (pos, voucher_number) values ($pos, $max_voucher_number);
but the first script set a sleep(10) before the insert in order to test lock for the sequence
The problem arise when I add a UNIQUE INDEX
ALTER TABLE `test_table`
ADD UNIQUE INDEX `per_pos_sequence` (`pos` ASC, `voucher_number` ASC);
Then I get this error for the :
SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when
trying to get lock; try restarting transaction
Why do I get that error if an index is present?
Is it possible to mantain the index and get no errors?
I'd guess you are running into the behavior described in this bug report: https://bugs.mysql.com/bug.php?id=98324
It matches an increase in deadlocks we have observed at my company since upgrading to MySQL 5.7.26 or later. Many different applications and tables have increased frequency of deadlocks, and the only thing in common is that they are using tables with PRIMARY KEY and also secondary UNIQUE KEY.
The response to the bug report says that deadlocks happening is not a bug, but a natural consequence of concurrent clients requesting locks. If the locks cannot be granted atomically, then there is a chance of deadlocks. Application clients should not treat this as a bug or an error. Just follow the instructions in the deadlock message: re-try the failed transaction.
The only ways I know of to avoid deadlocks are:
Avoid defining multiple unique keys on a given table.
Disallow concurrent clients requesting locks against the same table.
Use pessimistic table-locks to ensure clients access the table serially.
First, a bit of necessary background (please, bear with me). I work as a developers of a web application using MySQL for persistance. We have implemented audit logging by creating an audit trail table for each data table. We might for example have the following table definitions for a Customer entity:
-- Data table definition.
CREATE TABLE my_database.customers (
CustomerId INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
FirstName VARCHAR(255) NOT NULL,
LastName VARCHAR(255) NOT NULL,
-- More data columns, removed for simplicity.
...
);
-- Audit table definition in separate schema.
CREATE TABLE my_database_audittrail.customers (
CustomerId INT(11) DEFAULT NULL,
FirstName VARCHAR(255) DEFAULT NULL,
LastName VARCHAR(255) DEFAULT NULL,
-- More data columns.
...
-- Audit meta data columns.
ChangeTime DATETIME NOT NULL,
ChangeByUser VARCHAR(255) NOT NULL
);
As you can see, the audit table is simply a copy of the data table plus some metadata. Note that the audit table doesn't have any keys. When, for example, we update a customer, our ORM generates SQL similar to the following:
-- Insert a copy of the customer entity, before the update, into the audit table.
INSERT INTO my_database_audittrail.customers (
CustomerId,
FirstName,
LastName,
...
ChangeTime,
ChangeByUser)
)
SELECT
CustomerId,
FirstName,
LastName,
...
NOW(),
#ChangeByUser
FROM my_database.customers
WHERE CustomerId = #CustomerId;
-- Then update the data table.
UPDATE
my_database.customers
SET
FirstName = #FirstName,
LastName = #LastName,
...
WHERE CustomerId = #CustomerId;
This has worked well enough. Recently, however, we needed to add a primary key column to the audit tables for various reasons, changing the audit table definition to something similar to the following:
CREATE TABLE my_database_audittrail.customers (
__auditId INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
CustomerId INT(11) DEFAULT NULL,
FirstName VARCHAR(255) DEFAULT NULL,
LastName VARCHAR(255) DEFAULT NULL,
...
ChangeTime DATETIME NOT NULL,
ChangeByUser VARCHAR(255) NOT NULL
);
The SQL generated by our ORM when updating data tables has not been modified. This change seem to have increased the risk of deadlock very much. The system in question is a web application with a number of nightly batch jobs. The increase in deadlocks doesn't show in the day to day use of the system by our web users. The nightly batch jobs, however, do suffer from the deadlocks very much as they do intense work on a few database tables. Our "solution" has been to add a retry-upon-deadlock strategy (hardly controversial) and while this seems to work fine I would very much like to understand why the above change has increased the risk of deadlocks that much (and if we can somehow remedy the problem).
Further information:
Our nightly batch jobs do INSERTS, UPDATES and DELETES on our data tables. Only INSERTS are performed on the audit tables.
We use repeatable read isolation level on out database transactions.
Before this change, we haven't seen a single deadlock when running our nightly batch jobs.
UPDATE: Checked SHOW ENGINE INNODB STATUS to determine the cause of the deadlocks and found this:
*** WAITING FOR THIS LOCK TO BE GRANTED:
TABLE LOCK table `my_database_audittrail`.`customers` trx id 24972756464 lock mode AUTO-INC waiting
I was under the impression that auto increments was handled outside of any transactions in order to avoid using the same auto increment value in different transactions? But I guess the AUTO_INCREMENT property on the primary key we introduced seems to be the problem?
This is speculation.
Inserting or updating into a table with indexes not only locks the data pages but also the index pages, including the higher levels of the index. When multiple threads are affecting records at the same time, they may lock different portions of the index.
This would not generally show up with single record inserts. However, two statements that are updating multiple records might start acquiring locks on the index and find that they are deadlocking each other. Retry may be sufficient for fixing this problem. Alternatively, it seems that "too much" may be running at one time and you may want to consider how the nightly update work is laid out.
When inserting into tables with auto increment columns, MySQL uses different strategies to acquire values for the auto increments column(s) depending on which type of insert is made, on your insert statements and how MySQL is configured to handle auto increment columns, an insert may result in a complete table lock.
With "simple inserts", i.e inserts where MySQL can determine before hand the number of rows which will be inserted into a table (e.g INSERT INTO table (col1, col2) VALUES (val1, val2);) auto increment column values are acquired using a light weight lock on the auto increment counter. This light weight lock is released as soon as the auto increment values are acquired so one won't have to wait until the actual insert to complete. I.e no table lock.
However, with "bulk inserts", where MySQL cannot determine the number of inserted rows before hand (e.g INSERT INTO table (col1, col2) SELECT col1, col2 FROM table2 WHERE ...;) a table lock is created to acquire auto increment column values and not relinquished until the insert is completed.
The above is per MySQL's default configuration. MySQL can be configured to not use table locks on bulk inserts but this may cause auto increment columns to have different values on masters and slaves (if replication is set up) and thus may or may not be an viable option.
I am struggling with INSERT .. ON DUPLICATE KEY UPDATE for a file on a big InnoDB table.
My values table saves the details for each entity belonging to an client. An entity can have only one value for a particular key. So when a change is happening we are updating the same. The table looks something like below:
CREATE TABLE `key_values` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`client_id` int(11) NOT NULL COMMENT 'customer/tenant id',
`key_id` int(11) NOT NULL COMMENT 'reference to the keys',
`entity_id` bigint(20) NOT NULL,
`value` text,
`modified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `client_id` (`client_id`,`entity_id`,`key_id`),
KEY `client_id_2` (`client_id`,`key_id`)
) ;
All writes queries are of the form:
INSERT INTO `key_values`
(client_id, key_id, entity_id,value)
values
(23, 47, 147, 'myValue'), (...), (...)...
ON DUPLICATE KEY UPDATE value = values(value);
The table is around 350M records by now and is growing pretty fast.
The writes to table can happen from real time integration often
inserting less than 10 rows or as a bulk of 25K from offline sources.
For a given client, only one bulk operation can run at a time. This is reduce the row locks between insert
Lock wait time out period is set at 50 seconds
Currently, when the offline activities are happening sometimes(not always) we are getting an lock wait time-out. What could be possible changes without to avoid the time out ?
A design change at the moment is not possible ( sharding/partitioning/cluster).
REPLACE is another candidate, but I dont want to give delete privilege in production to anything from code.
INSERT IGNORE and then UPDATE is a good candidate, but will it give much improvement?
What other options do I have?
Thanks in advance for all suggestion and answers.
Regarding the lock wait timeout, this can be changed via the mysql configuration setting innodb_lock_wait_timeout which can be modified dynamically (without restarting mysql), in addition to changing it in your my.cnf.
Regarding the lock waits, one thing to consider with mysql is the default transaction isolation level, which is REPEATABLE READ. The side effect of this setting is that much more locking occurs for reads that you might expect (especially if you had a SQL Server background, which has a default tran iso level of READ COMMITTED). Now, if you don't need REPEATABLE READ, you can change your tran iso level, either in a query, using the SET TRANSACTION ISOLATION LEVEL syntax, or for the whole server, using the config setting transaction-isolation. I recommend using READ COMMITTED, and consider if there are other places in your application where even 'dirtier' reads are acceptable (in which case you can use READ UNCOMMITTED.
I'm running MySql 5.0.22 and have a really unwieldy table containing approximately 5 million rows.
Some, but not all rows are referenced by a foreign key to another table.
All attempts to cull the unreferenced rows have failed so far, resulting in lock-timeouts every time.
Copying the rows I want to an alternate table also failed with lock-timeout.
Suspiciously, even a statement that should finish instantaneously like the one below will also fail with "lock timeout":
DELETE FROM mytable WHERE uid_pk = 1 LIMIT 1;
...it's at this point that I've run out of ideas.
Edit: For what it's worth, I've been working through this on my dev system, so only I am actually using the database at this moment so there shouldn't be any locking going on outside of the SQL I'm running.
Any MySql gurus out there have suggestions on how to tame this rogue table?
Edit #2: As requested, the table structure:
CREATE TABLE `tunknowncustomer` (
`UID_PK` int(11) NOT NULL auto_increment,
`UNKNOWNCUSTOMERGUID` varchar(36) NOT NULL,
`CREATIONDATE` datetime NOT NULL,
`EMAIL` varchar(100) default NULL,
`CUSTOMERUID` int(11) default NULL,
PRIMARY KEY (`UID_PK`),
KEY `IUNKNOWCUST_CUID` (`CUSTOMERUID`),
KEY `IUNKNOWCUST_UCGUID` (`UNKNOWNCUSTOMERGUID`),
CONSTRAINT `tunknowncustomer_ibfk_1` FOREIGN KEY (`CUSTOMERUID`) REFERENCES `tcustomer` (`UID_PK`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8$$
Note, attempting to drop the FK also times out.
I had the same problem with an innodb table. optimize table corrected it.
Ok, I finally found an approach that worked to trim the unwanted rows from my large InnoDB table! Here's how I did it:
Stopped using MySQL Workbench (they have a hard-coded execution timeout of 30 seconds)
Opened a command prompt
Renamed the "full" table using ALTER TABLE
Created an empty table using the original table name and structure
Rebooted MySQL
Turned OFF 'autocommit' with SET AUTOCOMMIT = 0
Deleted a limited number of rows at a time, ramping up my limit after each success
Did a COMMIT; in between delete statements since turning off autocommit really left me inside of one large transaction
The whole effort looked somewhat like this:
ALTER TABLE `ep411`.`tunknowncustomer` RENAME TO `ep411`.`tunknowncustomer2`;
...strange enough, renaming the table was the only ALTER TABLE command that would finish right away.
delimiter $$
CREATE TABLE `tunknowncustomer` (
...
) ENGINE=InnoDB DEFAULT CHARSET=utf8$$
...then a reboot just in case my previous failed attempts could block any new work done...
SET AUTOCOMMIT = 0;
delete from tunknowncustomer2 where customeruid is null limit 1000;
delete from tunknowncustomer2 where customeruid is null limit 100000;
commit;
delete from tunknowncustomer2 where customeruid is null limit 1000000;
delete from tunknowncustomer2 where customeruid is null limit 1000000;
commit;
...Once I got into deleting 100k at a time InnoDB's execution time dropped with each successful command. I assume InnoDB starts doing read-aheads on large scans. Doing commits would reset the read-ahead data, so I spaced out the COMMITs to every 2 million rows until the job was done.
I wrapped-up the task by copying the remaining rows into my "empty" clone table, then dropping the old (renamed) table.
Not a graceful solution, and it doesn't address any reasons why deleting even a single row from a large table should fail, but at least I got the result I was looking for!