Deadlocks - At what point are locks acquired in a transaction? - mysql

animals table
+----------+-------+
| name | value |
+----------+-------+
| Aardvark | 10 |
+----------+-------+
birds table
+---------+-------+
| name | value |
+---------+-------+
| Buzzard | 20 |
+---------+-------+
Session 1:
mysql> START TRANSACTION;
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT value FROM Animals WHERE name='Aardvark' FOR SHARE;
+-------+
| value |
+-------+
| 10 |
+-------+
1 row in set (0.00 sec)
Session 2:
mysql> START TRANSACTION;
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT value FROM Birds WHERE name='Buzzard' FOR SHARE;
+-------+
| value |
+-------+
| 20 |
+-------+
1 row in set (0.00 sec)
--waits to lock
mysql> UPDATE Animals SET value=30 WHERE name='Aardvark';
Session 1:
mysql> UPDATE Birds SET value=40 WHERE name='Buzzard';
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
Can we say that the lock is acquired UNTIL THE SQL SENTENCE IS REACHED or does a transaction obtain all locks before?
A lot of people say that we have to acquire all the (anticipated) locks that we are going to use throughout the transaction to avoid deadlocks, so the question arose in my mind.

Yes, row locks are acquired on demand, when you execute the SQL statement that requires those locks.
This means that if two sessions are running concurrently, they may acquire their respective locks in an interleaved fashion, not an atomic fashion. Therefore they might both want to lock a resource that is already locked by the other session, and in that case they create a deadlock.
One fix to prevent deadlocks is to acquire all the locks you need during the transaction — and more to the point, acquire the locks atomically. That is, all the locks must be acquired at once, and if that isn't successful, then release all the locks. Locks cannot be acquired in an interleaved fashion.
InnoDB doesn't really have a feature to do this. You may resort to table locking with MySQL's LOCK TABLES statement.
A different strategy is not to avoid deadlocks, but just recover from deadlocks when they occur.

Related

LOCK TABLES table WRITE blocks my readings

I am at the REPEATABLE-READ level.
Why does it make me wait?
I understand that all reads (SELECTs) at any level are non-blocking.
what am I missing?
Session 1:
mysql> lock tables users write;
Query OK, 0 rows affected (0.00 sec)
Session 2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from users where id = 1; // wait
Session 1:
mysql> unlock tables;
Query OK, 0 rows affected (0.00 sec)
Session 2:
mysql> select * from users where id = 1;
+----+-----------------+--------------------+------+---------------------+--------------------------------------------------------------+----------------+---------------------+---------------------+------------+
| id | name | email | rol | email_verified_at | password | remember_token | created_at | updated_at | deleted_at |
+----+-----------------+--------------------+------+---------------------+--------------------------------------------------------------+----------------+---------------------+---------------------+------------+
| 1 | Bella Lueilwitz | orlo19#example.com | NULL | 2022-08-01 17:22:29 | $2y$10$92IXUNpkjO0rOQ5byMi.Ye4oKoEa3Ro9llC/.og/at2.uheWG/igi | MvMlaX9TQj | 2022-08-01 17:22:29 | 2022-08-01 17:22:29 | NULL |
+----+-----------------+--------------------+------+---------------------+--------------------------------------------------------------+----------------+---------------------+---------------------+------------+
1 row in set (10.51 sec)
In this question the opposite is true
Why doesn't LOCK TABLES [table] WRITE prevent table reads?
You reference a question about MySQL 5.0 posted in 2013. The answer from that time suggests that the client was allowed to get a result that had been cached in the query cache. Since then, MySQL 5.6 and 5.7 disabled the query cache by default, and MySQL 8.0 removed the feature altogether. This is a good thing.
The documentation says:
WRITE lock:
Only the session that holds the lock can access the table. No other session can access it until the lock is released.
This was true in the MySQL 5.0 days too, but the query cache allowed some clients to get around it. But I guess it wasn't reliable even then, because if the client ran a query that happened not to be cached, I suppose it would revert to the documented behavior. Anyway, it's moot, because all currently supported versions of MySQL should have the query cache disabled or removed.

Does only running "BEGIN;" or "START TRANSACTION;" query really start a transaction? (MySQL)

On MySQL, first, I ran BEGIN; or START TRANSACTION; to start a transaction as shown below:
mysql> BEGIN;
Query OK, 0 rows affected (0.00 sec)
Or:
mysql> START TRANSACTION;
Query OK, 0 rows affected (0.00 sec)
Then, I ran the query below to check if the transaction is running but I got nothing as shown below:
mysql> SELECT trx_id FROM information_schema.innodb_trx;
Empty set (0.00 sec)
So just after this, I ran the query below to show all the rows of person table:
mysql> SELECT * FROM person;
+----+------+
| id | name |
+----+------+
| 1 | John |
| 2 | Tom |
+----+------+
2 rows in set (0.00 sec)
Then again, I ran the query below, then now, I could check the transaction is actually running:
mysql> SELECT trx_id FROM information_schema.innodb_trx;
+-----------------+
| trx_id |
+-----------------+
| 284321631771648 |
+-----------------+
1 row in set (0.00 sec)
So, does only running BEGIN; or START TRANSACTION; query really start a transaction?
Yes. start transaction starts a transaction. Otherwise, that statement would have a very misleading name.
You can check this by trying to do things you cannot do inside a transaction, e.g. change the isolation level:
START TRANSACTION;
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
ERROR 1568 (25001): Transaction characteristics can't be changed while a transaction is in progress
So why don't you see it in the innodb_trx table?
Because innodb_trx only lists transactions that involve InnoDB. Until you e.g. query an InnoDB-table, the InnoDB engine doesn't know about that transaction and doesn't list it, just as shown in your tests. Try changing the table to a MyISAM-table. It will not show up in innodb_trx even after you ran a query on it.
A transaction is started on the mysql level, not the storage engine level.
For example, the NDB engine also supports transactions and has its own transaction table where it only shows transactions that involve NDB tables.

Can MySQL UPDATE locks starve due to continuous SHARE locks?

I have 2 different transactions where one is using read locks (FOR SHARE) for SELECT statements and the other uses write locks (FOR UPDATE).
Let's say they are trying to acquire the lock on the same row. Here's the scenario I'm trying to understand what's happening.
Let's say I have continuous stream of requests using the read locks and occasionally I need to acquire the write lock.
Are these locks using FIFO strategy to avoid starvation or some other strategy such as read locks would be acquired as long as it can acquire the lock and write lock would wait all the reads to drain (even the new ones in this case).
I'm suspecting 2nd might be happening but I'm not 100% sure.
I'm investigating an issue and couldn't find a good documentation about this.
If you lack documentation, you can try an experiment:
Window 1:
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from tablename for share;
+---------------------+
| ldt |
+---------------------+
| 1969-12-31 16:00:00 |
+---------------------+
1 row in set (0.00 sec)
Window 2:
mysql> update tablename set ldt=now();
(hangs, waiting for lock)
Window 3:
mysql> select * from tablename for share;
(hangs, also waiting for lock)
This indicates that the X-lock request is blocking subsequent S-lock requests.
50 seconds passes, and then:
Window 2:
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
And then immediately:
Window 3:
mysql> select * from tablename for share;
+---------------------+
| ldt |
+---------------------+
| 1969-12-31 16:00:00 |
+---------------------+
1 row in set (41.14 sec)
The select in window 3 was blocked while waiting for the update in window 2. When the update timed out, then the select in window 3 was able to proceed.

How MVCC works with Lock in MySql?

I know that using the locks or MVCC in Mysql can achieve concurrency control, such as repeatable-reading. But I don't know how MVCC avoids phantom-reading. In other places, I learned that it is generally implemented through MVCC and Gap-Lock, but what I currently understand is that MVCC does not need locks, that is, both updates and deletions are implemented using undo-logs. If so, how do MVCC and the lock mechanism work together?
For example, to avoid phantom-reading, would MVCC add a gap-lock on some rows in T1? If so, how MVCC does when updates occurred in T2, just appends a update undo-log generally? or blocks it?
MySQL (specifically, InnoDB) does not support REPEATABLE-READ for locking statements. For example, UPDATE, DELETE or SELECT...FOR UPDATE. These statements always take locks on the most recently committed row version, as if the transaction isolation level were READ-COMMITTED.
You can observe this happening:
mysql> create table mytable (id int primary key, x int);
Query OK, 0 rows affected (0.05 sec)
mysql> insert into mytable values (1, 42);
Query OK, 1 row affected (0.02 sec)
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from mytable;
+----+------+
| id | x |
+----+------+
| 1 | 42 |
+----+------+
So far, so good. Now open a second window and update the value:
mysql> update mytable set x = 84;
Query OK, 1 row affected (0.03 sec)
Rows matched: 1 Changed: 1 Warnings: 0
Now back in the first window, a non-locking read still views the original value because of REPEATABLE-READ, but a locking read views the most recently committed version:
mysql> select * from mytable;
+----+------+
| id | x |
+----+------+
| 1 | 42 |
+----+------+
1 row in set (0.00 sec)
mysql> select * from mytable for update;
+----+------+
| id | x |
+----+------+
| 1 | 84 |
+----+------+
1 row in set (0.00 sec)
mysql> select * from mytable;
+----+------+
| id | x |
+----+------+
| 1 | 42 |
+----+------+
1 row in set (0.00 sec)
You can go back and forth as many times as you want, and the same transaction can return both values, depending on doing a locking read vs. non-locking read.
This is a strange behavior of InnoDB, but it allows reads to not be blocked. I have used other MVCC implementations such as InterBase/Firebird, which solve this differently. It would block the read until the transaction in the second window commits or rolls back. If it rolls back, then the locking read can read the original value. If the other transaction commits, then the locking read gets an error.
InnoDB makes a different choice on how to implement MVCC, to avoid blocking the read. But it causes the strange behavior where a locking read must view the latest committed row version.
As the song says, "you can't always get what you want."

MySQL InnoDB "SELECT FOR UPDATE" - SKIP LOCKED equivalent

Is there any way to skip "locked rows" when we make "SELECT FOR UPDATE" in MySQL with an InnoDB table?
E.g.: terminal t1
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select id from mytable ORDER BY id ASC limit 5 for update;
+-------+
| id |
+-------+
| 1 |
| 15 |
| 30217 |
| 30218 |
| 30643 |
+-------+
5 rows in set (0.00 sec)
mysql>
At the same time, terminal t2:
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select id from mytable where id>30643 order by id asc limit 2 for update;
+-------+
| id |
+-------+
| 30939 |
| 31211 |
+-------+
2 rows in set (0.01 sec)
mysql> select id from mytable order by id asc limit 5 for update;
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
mysql>
So if I launch a query forcing it to select other rows, it's fine.
But is there a way to skip the locked rows?
I guess this should be a redundant problem in the concurrent process, but I did not find any solution.
EDIT:
In reality, my different concurrent processes are doing something apparently really simple:
take the first rows (which don't contain a specific flag - e.g.: "WHERE myflag_inUse!=1").
Once I get the result of my "select for update", I update the flag and commit the rows.
So I just want to select the rows which are not already locked and where myflag_inUse!=1...
The following link helps me to understand why I get the timeout, but not how to avoid it:
MySQL 'select for update' behaviour
mysql> SHOW VARIABLES LIKE "%version%";
+-------------------------+-------------------------+
| Variable_name | Value |
+-------------------------+-------------------------+
| innodb_version | 5.5.46 |
| protocol_version | 10 |
| slave_type_conversions | |
| version | 5.5.46-0ubuntu0.14.04.2 |
| version_comment | (Ubuntu) |
| version_compile_machine | x86_64 |
| version_compile_os | debian-linux-gnu |
+-------------------------+-------------------------+
7 rows in set (0.00 sec)
MySQL 8.0 introduced support for both SKIP LOCKED and NO WAIT.
SKIP LOCKED is useful for implementing a job queue (a.k.a batch queue) so that you can skip over locks that are already locked by a concurrent transaction.
NO WAIT is useful for avoiding waiting until a concurrent transaction releases the locks that we are also interested in locking.
Without NO WAIT, we either have to wait until the locks are released (at commit or release time by the transaction that currently holds the locks) or the lock acquisition times out. NO WAIT acts as a lock timeout with a value of 0.
For more details about SKIP LOCK and NO WAIT.
This appears to now exist in MySQL starting in 8.0.1:
https://mysqlserverteam.com/mysql-8-0-1-using-skip-locked-and-nowait-to-handle-hot-rows/
Starting with MySQL 8.0.1 we are introducing the SKIP LOCKED modifier
which can be used to non-deterministically read rows from a table
while skipping over the rows which are locked. This can be used by
our booking system to skip orders which are pending. For example:
However, I think that version is not necessarily production ready.
Unfortunately, it seems that there is no way to skip the locked row in a select for update so far.
It would be great if we could use something like the Oracle 'FOR UPDATE SKIP LOCKED'.
In my case, the queries launched in parallel are both exactly the same, and contain a 'where' clause and a 'group by' on a several millions of rows...because the queries need between 20 and 40 seconds to run, that was (as I already knew) a big part of the problem.
The only -temporary and not the best- solution I saw was to move some (i.e.: millions of) rows that I would not (directly) use in order to reduce the time the query will take.
So I will still have the same behavior but I will wait less time...
I was expecting a way to not select the locked row in the select.
I don't mark this as an answer, so if a new clause from mysql is added (or discovered), I can accept it later...
I'm sorry, but I think you approach the problem from a wrong angle. If your user wants to list records from a table that satisfy certain selection criteria, then your query should return them all, or return with an error message and provide no resultset whatsoever. But the query should not reurn only a subset of the results leading the user to belive that he has all the matching records.
The issue should be addressed by making sure that your application locks as few rows as possible, for as little time as possible.
Walk through the table in chunks of the PRIMARY KEY, using some suitable LIMIT so you are not looking at "too many" rows at once.
By using the PK, you are ordering things in a predictable way; this virtually eliminates deadlocks.
By using LIMIT, you will keep from hogging too much at once. The LIMIT should be embodied as a range over the PK. This makes it quite clear if two threads are about to step on each other.
More details are (indirectly) in my blog on big deletes.