I have locked one row in one transaction by following query
START TRANSACTION;
SELECT id FROM children WHERE id=100 FOR UPDATE;
And in another transaction i have a query as below
START TRANSACTION;
SELECT id FROM children WHERE id IN (98,99,100) FOR UPDATE;
It gives error lock wait timeout exceeded.
Here 100 is already locked (in first transaction ) But the ids 98,99 are not locked.Is there any possibility return records of 98,99 if only 100 is row locked in above query.So result should be as below
Id
===
98
99
===
Id 100 should be ignored because 100 is locked by a transaction.
Looks like SKIP LOCKED option mentioned in a previous answer is now available in MySQL. It does not wait to acquire a row lock and allows you to work with rows that are not currently locked.
From MySQL 8.0.0 Release Notes/Changes in MySQL 8.0.1:
InnoDB now supports NOWAIT and SKIP LOCKED options with SELECT ... FOR SHARE and SELECT ... FOR UPDATE locking read statements. NOWAIT causes the statement to return immediately if a requested row is locked by another transaction. SKIP LOCKED removes locked rows from the result set. See Locking Read Concurrency with NOWAIT and SKIP LOCKED.
Sample usage (complete example with outputs can be found in the link above):
START TRANSACTION;
SELECT * FROM tableName FOR UPDATE SKIP LOCKED;
Also, it might be good to include the warning in the Reference Manual here as well:
Queries that skip locked rows return an inconsistent view of the data. SKIP LOCKED is therefore not suitable for general transactional work. However, it may be used to avoid lock contention when multiple sessions access the same queue-like table.
MySQL does not have a way to ignore locked rows in a SELECT. You'll have to find a different way to set a row aside as "already processed".
The simplest way is to lock the row briefly in the first query just to mark it as "already processed", then unlock it and lock it again for the rest of the processing - the second query will wait for the short "marker" query to complete, and you can add an explicit WHERE condition to ignore already-marked rows. If you don't want to rely on the first operation being able to complete successfully, you may need to add a bit more complexity with timestamps and such to clean up after those failed operations.
MySQL does not have this feature. For anyone searching for this topic in general, some RDBMS have better/smarter locking features than others.
For developers constrained to MySQL, the best approach is to add a column (or use an existing, e.g., status column) that can be set to "locked" or "in progress" or similar, execute a SELECT ID, * ... WHERE IN_PROGRESS != 1 FOR UPDATE; to get the row ID you want to lock, issue UPDATE .. SET IN_PROGRESS = 1 WHERE ID = XX to unlock the records.
Using LOCK IN SHARE MODE is almost never the solution because while it'll let you read the old value, but the old value is in the process of being updated so unless you are performing a non-atomic task, there's no point in even looking at that record.
Better* RDBMS recognize this pattern (select one row to work on and lock it, work on it, unlock it) and provide a smarter approach that lets you only search unlocked records. For example, PostgreSQL 9.5+ provide SELECT ... SKIP LOCKED which only selects from within the unlocked subset of rows matching the query. That lets you obtain an exclusive lock on a row, service that record to completion, then update & unlock the record in question without having to block other threads/consumers from being able to work independent of yourself.
*Here "better" means from the perspective of atomic updates, multi-consumer architecture, etc. and not necessarily "better designed" or "overall better." Not trying to start a flamewar here.
As per http://dev.mysql.com/doc/refman/5.0/en/innodb-locking-reads.html
The solution is to perform the SELECT in a locking mode using LOCK IN SHARE MODE:
SELECT * FROM parent WHERE NAME = 'Jones' LOCK IN SHARE MODE;
Related
I want to lock a specific row in MySQL (InnoDB), so that specific row can't be read by another connection.
start transaction;
Select apples from fruit where apples = 'golden';
commit;
Now I want to apply an exclusive lock
exclusive lock
A kind of lock that prevents any other transaction from locking the same row.
https://dev.mysql.com/doc/refman/5.5/en/innodb-locking.html#innodb-shared-exclusive-locks
I'm assuming, the lock get's applied, then the second connection trying to access the same row, has to wait till the first connection releases the lock with a "commit;"
My question is, how do I adjust my SQL Statement to apply that lock? Or does this automatically happen with the start transaction?
I've been trying to find a good example and haven't.
START TRANSACTION;
Select apples
from fruit
where apples = 'golden'
FOR UPDATE;
...
COMMIT;
The SELECT ... FOR UPDATE statement first locks the row, then returns it to you.
Anyone else trying to do the same thing to the same row at the same time will wait for your COMMIT or ROLLBACK and then they will receive the row... with any changes you made.
If you have no intention of changing the row, you just want to make sure nobody else does can, then you can SELECT ... LOCK IN SHARE MODE.
Note that, either way, it's technically not the "row," it's actually the index record that is being locked, but in InnoDB, "everything is an index" (even a table with no indexes at all does in fact still have one index, siently created by the system) so the net result is the same.
The Problem
I'm trying to figure out how to correctly set up a transaction in a database, and account for potential latency.
The Setup
In my example I have a table of users, keys, where each user can have multiple keys, and a config table that dictates how many keys each user is allowed to have.
I want to run a stored procedure that:
figures out if the given user is allowed to request a key.
get an available, unclaimed key .
attempts to redeem the key for the given user.
the pseudocode for the procedure would be:
START TRANSACTION
(1) CALL check_permission(...,#result);
IF (#result = 'has_permission') THEN
(2) SET #unclaimed_key_id = (QUERY FOR RETURNING AVAILABLE KEY ID);
(3) CALL claim_key(#unclaimed_key_id);
END IF;
COMMIT;
The problem that I am running into, is that when I simulate lag after step 1, (by using SELECT SLEEP(<seconds>)), it's possible for a given user to redeem multiple keys when they only have permissions to redeem one, by running the procedure in multiple sessions before the first procedure has finished its sleep (which again, is to simulate lag)
Here is the code for the Tables and the Procedures
(note: for the small example I didn't bother with indexes and foreign keys, but obviously I use those on the actual project).
To see my issue just set up the tables and procedures in a database, then open two mysql terminals, and in the first run this:
CALL `P_user_request_key`(10,1,#out);
SELECT #out;
And then quickly (you have 10 seconds) in the second run this:
CALL `P_user_request_key`(0,1,#out);
SELECT #out;
Both queries will successfully return key_claimed and User Bob will end up with 4 keys assigned to him, although the max value in config is set to 3 per user.
The Questions
What is the best way of avoiding issues like this? I'm trying to use a transaction but I feel like It's not going to help specifically with this issue, and may be implementing this wrong.
I realize that one possible way to fix the problem would be to just encapsulate everything in one large update query, but I would prefer to avoid that, since I like being able to set up individual procedures, where each is only meant to do a single task.
The database behind this example is intended to be used by many (thousands) of concurrent users. As such it would be best if one user attempting to redeem a code doesn't block all other users from redeeming one. I'm fine with changing my code to just attempt to redeem again if another user already claimed a key, but it should absolutely not happen that a user can redeem two codes when they only have permission to get one.
You're off the hook for not wanting to encapsulate everything in one large query, because that won't actually solve anything either, it just makes it less likely.
What you need are locks on the rows, or locks on the index where the new row would be inserted.
InnoDB uses an algorithm called next-key locking that combines index-row locking with gap locking. InnoDB performs row-level locking in such a way that when it searches or scans a table index, it sets shared or exclusive locks on the index records it encounters. Thus, the row-level locks are actually index-record locks. In addition, a next-key lock on an index record also affects the “gap” before that index record. That is, a next-key lock is an index-record lock plus a gap lock on the gap preceding the index record. If one session has a shared or exclusive lock on record R in an index, another session cannot insert a new index record in the gap immediately before R in the index order.
http://dev.mysql.com/doc/refman/5.5/en/innodb-next-key-locking.html
So how do we get exclusive locks?
Two connections, mysql1 and mysql2, each of them requesting an exclusive lock using SELECT ... FOR UPDATE. The table 'history' has a column 'user_id' which is indexed. (It's also a foreign key.) There are no rows found, so they both appear to proceed normally as if nothing unusual is going to happen. The user_id 2808 is valid but has nothing in history.
mysql1> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql2> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql1> select * from history where user_id = 2808 for update;
Empty set (0.00 sec)
mysql2> select * from history where user_id = 2808 for update;
Empty set (0.00 sec)
mysql1> insert into history(user_id) values (2808);
... and I don't get my prompt back ... no response ... because another session has a lock, too ... but then:
mysql2> insert into history(user_id) values (2808);
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
Then mysql1 immediately returns success on the insert.
Query OK, 1 row affected (3.96 sec)
All that is left is for mysql1 to COMMIT and magically, we prevented a user with 0 entries from inserting more than 1 entry. The deadlock occurred because both sessions needed incompatible things to happen: mysql1 needed mysql2 to release its lock before it would be able to commit and mysql2 needed mysql1 to release its lock before it would be able to insert. Somebody has to lose that fight, and generally the thread that has done the least work is the loser.
But what if there had been 1 or more rows already existing when I did the SELECT ... FOR UPDATE? In that case, the lock would have been on the rows, so the second session to try to SELECT would actually block waiting for the SELECT until the first session decided to either COMMIT or ROLLBACK, at which time the second session would have seen an accurate count of the number of rows (including any inserted or deleted by the first session) and could have accurately decided the user already had the maximum allowed.
You can't outrace a race condition, but you can lock them out.
Please help me understand the use-case behind SELECT ... FOR UPDATE.
Question 1: Is the following a good example of when SELECT ... FOR UPDATE should be used?
Given:
rooms[id]
tags[id, name]
room_tags[room_id, tag_id]
room_id and tag_id are foreign keys
The application wants to list all rooms and their tags, but needs to differentiate between rooms with no tags versus rooms that have been removed. If SELECT ... FOR UPDATE is not used, what could happen is:
Initially:
rooms contains [id = 1]
tags contains [id = 1, name = 'cats']
room_tags contains [room_id = 1, tag_id = 1]
Thread 1: SELECT id FROM rooms;
returns [id = 1]
Thread 2: DELETE FROM room_tags WHERE room_id = 1;
Thread 2: DELETE FROM rooms WHERE id = 1;
Thread 2: [commits the transaction]
Thread 1: SELECT tags.name FROM room_tags, tags WHERE room_tags.room_id = 1 AND tags.id = room_tags.tag_id;
returns an empty list
Now Thread 1 thinks that room 1 has no tags, but in reality the room has been removed. To solve this problem, Thread 1 should SELECT id FROM rooms FOR UPDATE, thereby preventing Thread 2 from deleting from rooms until Thread 1 is done. Is that correct?
Question 2: When should one use SERIALIZABLE transaction isolation versus READ_COMMITTED with SELECT ... FOR UPDATE?
Answers are expected to be portable (not database-specific). If that's not possible, please explain why.
The only portable way to achieve consistency between rooms and tags and making sure rooms are never returned after they had been deleted is locking them with SELECT FOR UPDATE.
However in some systems locking is a side effect of concurrency control, and you achieve the same results without specifying FOR UPDATE explicitly.
To solve this problem, Thread 1 should SELECT id FROM rooms FOR UPDATE, thereby preventing Thread 2 from deleting from rooms until Thread 1 is done. Is that correct?
This depends on the concurrency control your database system is using.
MyISAM in MySQL (and several other old systems) does lock the whole table for the duration of a query.
In SQL Server, SELECT queries place shared locks on the records / pages / tables they have examined, while DML queries place update locks (which later get promoted to exclusive or demoted to shared locks). Exclusive locks are incompatible with shared locks, so either SELECT or DELETE query will lock until another session commits.
In databases which use MVCC (like Oracle, PostgreSQL, MySQL with InnoDB), a DML query creates a copy of the record (in one or another way) and generally readers do not block writers and vice versa. For these databases, a SELECT FOR UPDATE would come handy: it would lock either SELECT or the DELETE query until another session commits, just as SQL Server does.
When should one use REPEATABLE_READ transaction isolation versus READ_COMMITTED with SELECT ... FOR UPDATE?
Generally, REPEATABLE READ does not forbid phantom rows (rows that appeared or disappeared in another transaction, rather than being modified)
In Oracle and earlier PostgreSQL versions, REPEATABLE READ is actually a synonym for SERIALIZABLE. Basically, this means that the transaction does not see changes made after it has started. So in this setup, the last Thread 1 query will return the room as if it has never been deleted (which may or may not be what you wanted). If you don't want to show the rooms after they have been deleted, you should lock the rows with SELECT FOR UPDATE
In InnoDB, REPEATABLE READ and SERIALIZABLE are different things: readers in SERIALIZABLE mode set next-key locks on the records they evaluate, effectively preventing the concurrent DML on them. So you don't need a SELECT FOR UPDATE in serializable mode, but do need them in REPEATABLE READ or READ COMMITED.
Note that the standard on isolation modes does prescribe that you don't see certain quirks in your queries but does not define how (with locking or with MVCC or otherwise).
When I say "you don't need SELECT FOR UPDATE" I really should have added "because of side effects of certain database engine implementation".
Short answers:
Q1: Yes.
Q2: Doesn't matter which you use.
Long answer:
A select ... for update will (as it implies) select certain rows but also lock them as if they have already been updated by the current transaction (or as if the identity update had been performed). This allows you to update them again in the current transaction and then commit, without another transaction being able to modify these rows in any way.
Another way of looking at it, it is as if the following two statements are executed atomically:
select * from my_table where my_condition;
update my_table set my_column = my_column where my_condition;
Since the rows affected by my_condition are locked, no other transaction can modify them in any way, and hence, transaction isolation level makes no difference here.
Note also that transaction isolation level is independent of locking: setting a different isolation level doesn't allow you to get around locking and update rows in a different transaction that are locked by your transaction.
What transaction isolation levels do guarantee (at different levels) is the consistency of data while transactions are in progress.
What is SELECT FOR UPDATE?
SELECT FOR UPDATE is a SQL command that’s useful in the context of transactional workloads. It allows you to “lock” the rows returned by a SELECT query until the entire transaction that query is part of has been committed. Other transactions attempting to access those rows are placed into a time-based queue to wait, and are executed chronologically after the first transaction is completed.
BEGIN;
SELECT * FROM kv WHERE k = 1 FOR UPDATE;
UPDATE kv SET v = v + 5 WHERE k = 1;
COMMIT
I'm fairly sure this has a simple solution, but I haven't been able to find it so far. Provided an InnoDB MySQL database with the isolation level set to SERIALIZABLE, and given the following operation:
BEGIN WORK;
SELECT * FROM users WHERE userID=1;
UPDATE users SET credits=100 WHERE userID=1;
COMMIT;
I would like to make sure that as soon as the select inside the transaction is issued, the row corresponding to userID=1 is locked for reads until the transaction is done. As it stands now, UPDATEs to this row will wait for the transaction to be finished if it is in process, but SELECTs simply will read the previous value. I understand this is the expected behaviour in this case, but I wonder if there is a way to lock the row in such a way that SELECTs will also wait until the transaction is finished to return the values?
The reason I'm looking for that is that at some point, and with enough concurrent users, it could happen that while the previous transaction is in process someone else reads the "credits" to calculate something else. Ideally the code run by that someone else should wait for the transaction to finish to use the new value, because otherwise it could lead to irreversible desync issues.
Note that I don't want to lock the entire table for reads, just the specific row.
Also, I could add a boolean "locked" field to the tables and set it to 1 every time I'm starting a transaction but I don't really feel this is the most elegant solution here, unless there is absolutely no other way to handle this through mysql directly.
I found a workaround, specifically:
SELECT ... LOCK IN SHARE MODE sets a shared mode lock on the rows
read. A shared mode lock enables other sessions to read the rows but
not to modify them. The rows read are the latest available, so if they
belong to another transaction that has not yet committed, the read
blocks until that transaction ends.
(Source)
It seems that one can just include LOCK IN SHARE MODE in the critical SELECT statements that rely on transactional data and they will indeed wait for current transactions to finish before retrieving the row/s. For this to work the transaction has to use FOR UPDATE explicitly (as opposed to the original example I gave). E.g., given the following:
BEGIN WORK;
SELECT * FROM users WHERE userID=1 FOR UPDATE;
UPDATE users SET credits=100 WHERE userID=1;
COMMIT;
Anywhere else in the code I could use:
SELECT * FROM users WHERE userID=1 LOCK IN SHARE MODE;
Since this statement is not wrapped in a transaction, the lock is released immediately, thus having no impacts in subsequent queries, but if the row involving userID=1 has been selected for update within a transaction this statement would wait until the transaction is done, which is exactly what I was looking for.
You could try the SELECT ... FOR UPDATE locking read.
A SELECT ... FOR UPDATE reads the latest available data, setting exclusive locks on each row it reads. Thus, it sets the same locks a searched SQL UPDATE would set on the rows.
Please go through the following site: http://dev.mysql.com/doc/refman/5.0/en/innodb-locking-reads.html
I have a table of promo_codes that can be activated by a web application. There is a state column which can be either 0 for unactivated or 1 for activated. If I run a transaction with
SELECT FROM promo_codes WHERE state=0 LIMIT 1 FOR UPDATE;
UPDATE promo_codes SET state=1 WHERE id = ?;
What happens to a second transaction running:
SELECT FROM promo_codes WHERE state=0 LIMIT 1 FOR UPDATE;
Does it simply return the next row, or does it block until the first transaction is done?
I've actually started thinking about just setting a lock based on the row id in redis because it's obvious to me how that would work and I know it wouldn't create any performance issues in MySQL, but on the other hand, there must be a clean and performant way to make this work purely in SQL. Maybe I could use just do an UPDATE ... LIMIT 1 first, but how do I get the id of the promo code back in that case?
SELECT for UPDATE and LOCK IN SHARE MODE modifiers effectively run in READ-COMMITTED isolation mode even if current isolation mode is REPEATABLE-READ. This is done beause Innodb can only lock current version of row. Think about similar case and row being deleted. Even if Innodb would be able to set locks on rows which no more exist – would it do any good for you ? Not really – for example you could try to update the row which you just locked with SELECT FOR UPDATE but this row is already gone so you would get quite unexpected error updating the row which you thought you locked successfully. Anyway it is done this way for good all other decisions would be even more troublesome.
LOCK IN SHARE MODE is actually often used to bypass multiversioning and make sure we’re reading most current data, plus to ensure it can’t be changed. This for example can be used to read set of the rows, compute new values for some of them and write them back. If we would not use LOCK IN SHARE MODE we could be in trouble as rows could be update before we write new values to them and such update could be lost.