I am at the REPEATABLE-READ level.
Why does it make me wait?
I understand that all reads (SELECTs) at any level are non-blocking.
what am I missing?
Session 1:
mysql> lock tables users write;
Query OK, 0 rows affected (0.00 sec)
Session 2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from users where id = 1; // wait
Session 1:
mysql> unlock tables;
Query OK, 0 rows affected (0.00 sec)
Session 2:
mysql> select * from users where id = 1;
+----+-----------------+--------------------+------+---------------------+--------------------------------------------------------------+----------------+---------------------+---------------------+------------+
| id | name | email | rol | email_verified_at | password | remember_token | created_at | updated_at | deleted_at |
+----+-----------------+--------------------+------+---------------------+--------------------------------------------------------------+----------------+---------------------+---------------------+------------+
| 1 | Bella Lueilwitz | orlo19#example.com | NULL | 2022-08-01 17:22:29 | $2y$10$92IXUNpkjO0rOQ5byMi.Ye4oKoEa3Ro9llC/.og/at2.uheWG/igi | MvMlaX9TQj | 2022-08-01 17:22:29 | 2022-08-01 17:22:29 | NULL |
+----+-----------------+--------------------+------+---------------------+--------------------------------------------------------------+----------------+---------------------+---------------------+------------+
1 row in set (10.51 sec)
In this question the opposite is true
Why doesn't LOCK TABLES [table] WRITE prevent table reads?
You reference a question about MySQL 5.0 posted in 2013. The answer from that time suggests that the client was allowed to get a result that had been cached in the query cache. Since then, MySQL 5.6 and 5.7 disabled the query cache by default, and MySQL 8.0 removed the feature altogether. This is a good thing.
The documentation says:
WRITE lock:
Only the session that holds the lock can access the table. No other session can access it until the lock is released.
This was true in the MySQL 5.0 days too, but the query cache allowed some clients to get around it. But I guess it wasn't reliable even then, because if the client ran a query that happened not to be cached, I suppose it would revert to the documented behavior. Anyway, it's moot, because all currently supported versions of MySQL should have the query cache disabled or removed.
Related
On MySQL, first, I ran BEGIN; or START TRANSACTION; to start a transaction as shown below:
mysql> BEGIN;
Query OK, 0 rows affected (0.00 sec)
Or:
mysql> START TRANSACTION;
Query OK, 0 rows affected (0.00 sec)
Then, I ran the query below to check if the transaction is running but I got nothing as shown below:
mysql> SELECT trx_id FROM information_schema.innodb_trx;
Empty set (0.00 sec)
So just after this, I ran the query below to show all the rows of person table:
mysql> SELECT * FROM person;
+----+------+
| id | name |
+----+------+
| 1 | John |
| 2 | Tom |
+----+------+
2 rows in set (0.00 sec)
Then again, I ran the query below, then now, I could check the transaction is actually running:
mysql> SELECT trx_id FROM information_schema.innodb_trx;
+-----------------+
| trx_id |
+-----------------+
| 284321631771648 |
+-----------------+
1 row in set (0.00 sec)
So, does only running BEGIN; or START TRANSACTION; query really start a transaction?
Yes. start transaction starts a transaction. Otherwise, that statement would have a very misleading name.
You can check this by trying to do things you cannot do inside a transaction, e.g. change the isolation level:
START TRANSACTION;
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
ERROR 1568 (25001): Transaction characteristics can't be changed while a transaction is in progress
So why don't you see it in the innodb_trx table?
Because innodb_trx only lists transactions that involve InnoDB. Until you e.g. query an InnoDB-table, the InnoDB engine doesn't know about that transaction and doesn't list it, just as shown in your tests. Try changing the table to a MyISAM-table. It will not show up in innodb_trx even after you ran a query on it.
A transaction is started on the mysql level, not the storage engine level.
For example, the NDB engine also supports transactions and has its own transaction table where it only shows transactions that involve NDB tables.
I know that using the locks or MVCC in Mysql can achieve concurrency control, such as repeatable-reading. But I don't know how MVCC avoids phantom-reading. In other places, I learned that it is generally implemented through MVCC and Gap-Lock, but what I currently understand is that MVCC does not need locks, that is, both updates and deletions are implemented using undo-logs. If so, how do MVCC and the lock mechanism work together?
For example, to avoid phantom-reading, would MVCC add a gap-lock on some rows in T1? If so, how MVCC does when updates occurred in T2, just appends a update undo-log generally? or blocks it?
MySQL (specifically, InnoDB) does not support REPEATABLE-READ for locking statements. For example, UPDATE, DELETE or SELECT...FOR UPDATE. These statements always take locks on the most recently committed row version, as if the transaction isolation level were READ-COMMITTED.
You can observe this happening:
mysql> create table mytable (id int primary key, x int);
Query OK, 0 rows affected (0.05 sec)
mysql> insert into mytable values (1, 42);
Query OK, 1 row affected (0.02 sec)
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from mytable;
+----+------+
| id | x |
+----+------+
| 1 | 42 |
+----+------+
So far, so good. Now open a second window and update the value:
mysql> update mytable set x = 84;
Query OK, 1 row affected (0.03 sec)
Rows matched: 1 Changed: 1 Warnings: 0
Now back in the first window, a non-locking read still views the original value because of REPEATABLE-READ, but a locking read views the most recently committed version:
mysql> select * from mytable;
+----+------+
| id | x |
+----+------+
| 1 | 42 |
+----+------+
1 row in set (0.00 sec)
mysql> select * from mytable for update;
+----+------+
| id | x |
+----+------+
| 1 | 84 |
+----+------+
1 row in set (0.00 sec)
mysql> select * from mytable;
+----+------+
| id | x |
+----+------+
| 1 | 42 |
+----+------+
1 row in set (0.00 sec)
You can go back and forth as many times as you want, and the same transaction can return both values, depending on doing a locking read vs. non-locking read.
This is a strange behavior of InnoDB, but it allows reads to not be blocked. I have used other MVCC implementations such as InterBase/Firebird, which solve this differently. It would block the read until the transaction in the second window commits or rolls back. If it rolls back, then the locking read can read the original value. If the other transaction commits, then the locking read gets an error.
InnoDB makes a different choice on how to implement MVCC, to avoid blocking the read. But it causes the strange behavior where a locking read must view the latest committed row version.
As the song says, "you can't always get what you want."
Is there any way to skip "locked rows" when we make "SELECT FOR UPDATE" in MySQL with an InnoDB table?
E.g.: terminal t1
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select id from mytable ORDER BY id ASC limit 5 for update;
+-------+
| id |
+-------+
| 1 |
| 15 |
| 30217 |
| 30218 |
| 30643 |
+-------+
5 rows in set (0.00 sec)
mysql>
At the same time, terminal t2:
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select id from mytable where id>30643 order by id asc limit 2 for update;
+-------+
| id |
+-------+
| 30939 |
| 31211 |
+-------+
2 rows in set (0.01 sec)
mysql> select id from mytable order by id asc limit 5 for update;
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
mysql>
So if I launch a query forcing it to select other rows, it's fine.
But is there a way to skip the locked rows?
I guess this should be a redundant problem in the concurrent process, but I did not find any solution.
EDIT:
In reality, my different concurrent processes are doing something apparently really simple:
take the first rows (which don't contain a specific flag - e.g.: "WHERE myflag_inUse!=1").
Once I get the result of my "select for update", I update the flag and commit the rows.
So I just want to select the rows which are not already locked and where myflag_inUse!=1...
The following link helps me to understand why I get the timeout, but not how to avoid it:
MySQL 'select for update' behaviour
mysql> SHOW VARIABLES LIKE "%version%";
+-------------------------+-------------------------+
| Variable_name | Value |
+-------------------------+-------------------------+
| innodb_version | 5.5.46 |
| protocol_version | 10 |
| slave_type_conversions | |
| version | 5.5.46-0ubuntu0.14.04.2 |
| version_comment | (Ubuntu) |
| version_compile_machine | x86_64 |
| version_compile_os | debian-linux-gnu |
+-------------------------+-------------------------+
7 rows in set (0.00 sec)
MySQL 8.0 introduced support for both SKIP LOCKED and NO WAIT.
SKIP LOCKED is useful for implementing a job queue (a.k.a batch queue) so that you can skip over locks that are already locked by a concurrent transaction.
NO WAIT is useful for avoiding waiting until a concurrent transaction releases the locks that we are also interested in locking.
Without NO WAIT, we either have to wait until the locks are released (at commit or release time by the transaction that currently holds the locks) or the lock acquisition times out. NO WAIT acts as a lock timeout with a value of 0.
For more details about SKIP LOCK and NO WAIT.
This appears to now exist in MySQL starting in 8.0.1:
https://mysqlserverteam.com/mysql-8-0-1-using-skip-locked-and-nowait-to-handle-hot-rows/
Starting with MySQL 8.0.1 we are introducing the SKIP LOCKED modifier
which can be used to non-deterministically read rows from a table
while skipping over the rows which are locked. This can be used by
our booking system to skip orders which are pending. For example:
However, I think that version is not necessarily production ready.
Unfortunately, it seems that there is no way to skip the locked row in a select for update so far.
It would be great if we could use something like the Oracle 'FOR UPDATE SKIP LOCKED'.
In my case, the queries launched in parallel are both exactly the same, and contain a 'where' clause and a 'group by' on a several millions of rows...because the queries need between 20 and 40 seconds to run, that was (as I already knew) a big part of the problem.
The only -temporary and not the best- solution I saw was to move some (i.e.: millions of) rows that I would not (directly) use in order to reduce the time the query will take.
So I will still have the same behavior but I will wait less time...
I was expecting a way to not select the locked row in the select.
I don't mark this as an answer, so if a new clause from mysql is added (or discovered), I can accept it later...
I'm sorry, but I think you approach the problem from a wrong angle. If your user wants to list records from a table that satisfy certain selection criteria, then your query should return them all, or return with an error message and provide no resultset whatsoever. But the query should not reurn only a subset of the results leading the user to belive that he has all the matching records.
The issue should be addressed by making sure that your application locks as few rows as possible, for as little time as possible.
Walk through the table in chunks of the PRIMARY KEY, using some suitable LIMIT so you are not looking at "too many" rows at once.
By using the PK, you are ordering things in a predictable way; this virtually eliminates deadlocks.
By using LIMIT, you will keep from hogging too much at once. The LIMIT should be embodied as a range over the PK. This makes it quite clear if two threads are about to step on each other.
More details are (indirectly) in my blog on big deletes.
Mysql (5.5) Innodb in this certain case is putting table lock rather than row locks.
And this is causing failure of other insert queries to the table. Also this is a part of a larger transaction.
Insert into table x(x1,x2)
Select y1,y2 from y
where 'big sql case based conditions'
Now the select query select only part of table (based on which user) and not full table.
But mysql innodb is putting table locks.
Is there any way I can avoid this? Any help will be appreciated.
I think you are using tx in REPEATABLE READ mode. could you check out this?
mysql> show session variables like '%isol%';
+---------------+-----------------+
| Variable_name | Value |
+---------------+-----------------+
| tx_isolation | REPEATABLE-READ |
+---------------+-----------------+
If so, change it to 'READ COMMITTED' like this:
mysql> set session transaction isolation level read committed;
Query OK, 0 rows affected (0.00 sec)
mysql> show session variables like '%isol%';
+---------------+----------------+
| Variable_name | Value |
+---------------+----------------+
| tx_isolation | READ-COMMITTED |
+---------------+----------------+
Then, client A starts INSERT INTO .. SELECT and insert a row from client B. I think client B's INSERT would succeed.
As per the MySql documentation, MySql supports Multiple granularity locking(MGL).
case-1
Opened terminal-1:
// connected to mysql
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select id, status from tracking_number limit 5 for update;
+----+--------+
| id | status |
+----+--------+
| 1 | 0 |
| 2 | 0 |
| 3 | 0 |
| 4 | 0 |
| 5 | 0 |
+----+--------+
5 rows in set (0.00 sec)
mysql>
left it opened and opened terminal-2:
// connected to mysql
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select id, status from tracking_number limit 5 for update;
<!-- Hangs here. and after some time it says-->
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
Though there are plenty of rows to retrieve, T2 waits until t1 completes.
case-2
Left terminal-1 as is.Now in terminal-2:
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
<!-- case 2.1 -->
mysql> select id, status from tracking_number where id=1;
+----+--------+
| id | status |
+----+--------+
| 1 | 0 |
+----+--------+
1 row in set (0.00 sec)
mysql> select id, status from tracking_number where id=2;
+----+--------+
| id | status |
+----+--------+
| 2 | 0 |
+----+--------+
1 row in set (0.00 sec)
<!-- case 2.2 -->
mysql> select * from tracking_number where id=2 for update;
<!-- Hangs here. and after some time -->
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
But why in case 1, T2 waits for the same set of rows that T1 has locked?
Does it mean the unbounded select query (even with limint parameter. I have tried with different range also) blocks the entire table?
Is there any way to let transactions to lock independently without specifying the field of the record(i.e., without using where field=value)?
Generally (or as per Java concurrent locking), write lock is exclusive and read is not. In case 2.1, though the records are in write lock mode, how T2 can read the same records? Since this is allowed what is the point in locking it?
Case 2.2 is understood.
Opened a terminal and a transaction:
mysql> update tracking_number set status=4 where status=0 limit 5;
Query OK, 5 rows affected (0.00 sec)
Rows matched: 5 Changed: 5 Warnings: 0
Left it there and opened another terminal and transaction:
mysql> update tracking_number set status=5 where status=0 limit 5;
T2 did not succeed until i committed (or rollback) T1.
Why is this behavior?
Let me go through your cases and explain how these locks work:
1 case
T1 wants to update some rows in your test table. This transaction puts IX lock on all table and X lock on the first 5 rows.
T2 wants to update some rows in your test table. This transaction puts IX (because IX compatible with IX) lock on all table and tries to first 5 rows but it can't do it because X is not compatible with X
So we are fine.
2.1 case
T1 wants to update some rows in your test table. This transaction put IX lock on all table and X lock on the first 5 rows.
T2 wants to select some rows from your test table. And it does not place any locks (because InnoDB provides non-locking reads)
2.1 case
T1 wants to update some rows in your test table. This transaction put IX lock on all table and X lock on the first 5 rows.
T2 wants to update (select for update)some rows from your test table. Place IS on the whole table and tries to get S lock on the row and fails because X and S are uncompatible.
Also always be aware of isolation level: different level cause different mechanism to free/acquire locks
Hope it helps