Force-release a user-level lock in MySQL/MAriaDB - mysql

My session management (Zebra Session) uses user-level locks to avoid race conditions between two requests in the same session. To start the session, GET_LOCK is used. After closing the session, RELEASE_LOCK is used.
MariaDB [planner_20201026]> select GET_LOCK('session_ebe210e9b39f1ad3a409763be60efebff587aaaa', '5');
+-------------------------------------------------------------------+
| GET_LOCK('session_ebe210e9b39f1ad3a409763be60efebff587aaaa', '5') |
+-------------------------------------------------------------------+
| 1 |
+-------------------------------------------------------------------+
1 row in set (0.000 sec)
MariaDB [planner_20201026]> select RELEASE_LOCK('session_ebe210e9b39f1ad3a409763be60efebff587aaa');
+-----------------------------------------------------------------+
| RELEASE_LOCK('session_ebe210e9b39f1ad3a409763be60efebff587aaa') |
+-----------------------------------------------------------------+
| NULL |
+-----------------------------------------------------------------+
1 row in set (0.000 sec)
Now I am in a situation because of a reason which I do not know yet where the lock was not released properly. GET_LOCK finishes because of the timeout, RELEASE_LOCK tells me that it cannot release the lock because it was (according to the documentation) established by another thread:
MariaDB [xyz]> select GET_LOCK('session_ebe210e9b39f1ad3a409763be60efebff587ac8b', '5');
+-------------------------------------------------------------------+
| GET_LOCK('session_ebe210e9b39f1ad3a409763be60efebff587ac8b', '5') |
+-------------------------------------------------------------------+
| 0 |
+-------------------------------------------------------------------+
1 row in set (5.015 sec)
MariaDB [xyz]> select RELEASE_LOCK('session_ebe210e9b39f1ad3a409763be60efebff587ac8b');
+------------------------------------------------------------------+
| RELEASE_LOCK('session_ebe210e9b39f1ad3a409763be60efebff587ac8b') |
+------------------------------------------------------------------+
| 0 |
+------------------------------------------------------------------+
1 row in set (0.000 sec)
The session is now more or less blocked/useless/doomed, each request takes TIMEOUT seconds extra.
Is there any chance how I can clear that lock, especially after a timeout?

You can only use RELEASE_LOCK() to release a lock acquired in the same thread. A thread has no privilege to force another thread to give up its lock.
That would be a pretty useless locking system if you could acquire a lock but any other thread could unilaterally force you to release it!
One way you could work around this is to call IS_USED_LOCK() to tell you which thread holds the lock. It returns the integer thread id of the holder, or NULL if the lock is not held by anyone.
Then if you have SUPER privilege, your thread can KILL that other thread, and this will force it to release its lock (as well as disconnecting that client). But that's a pretty rude thing to do.
I have a feeling this is an XY Problem. You are searching for a solution to force locks held by other threads to be released, but this is a bad solution because it doesn't solve your real problem.
The real problem is:
Now I am in a situation because of a reason which I do not know yet where the lock was not released properly.
You need to think harder about this and design a system where you do not lose track of who has acquired the lock.
Hint: GET_LOCK(name, 0) may help. This returns immediately (that is, with zero seconds of timeout). If the lock can be acquired, it is acquired, and the return value of GET_LOCK is 1. If it was already held by another thread, the GET_LOCK still returns immediately, but with a return value of 0, telling you that it could not be acquired.

Related

Can MySQL UPDATE locks starve due to continuous SHARE locks?

I have 2 different transactions where one is using read locks (FOR SHARE) for SELECT statements and the other uses write locks (FOR UPDATE).
Let's say they are trying to acquire the lock on the same row. Here's the scenario I'm trying to understand what's happening.
Let's say I have continuous stream of requests using the read locks and occasionally I need to acquire the write lock.
Are these locks using FIFO strategy to avoid starvation or some other strategy such as read locks would be acquired as long as it can acquire the lock and write lock would wait all the reads to drain (even the new ones in this case).
I'm suspecting 2nd might be happening but I'm not 100% sure.
I'm investigating an issue and couldn't find a good documentation about this.
If you lack documentation, you can try an experiment:
Window 1:
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from tablename for share;
+---------------------+
| ldt |
+---------------------+
| 1969-12-31 16:00:00 |
+---------------------+
1 row in set (0.00 sec)
Window 2:
mysql> update tablename set ldt=now();
(hangs, waiting for lock)
Window 3:
mysql> select * from tablename for share;
(hangs, also waiting for lock)
This indicates that the X-lock request is blocking subsequent S-lock requests.
50 seconds passes, and then:
Window 2:
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
And then immediately:
Window 3:
mysql> select * from tablename for share;
+---------------------+
| ldt |
+---------------------+
| 1969-12-31 16:00:00 |
+---------------------+
1 row in set (41.14 sec)
The select in window 3 was blocked while waiting for the update in window 2. When the update timed out, then the select in window 3 was able to proceed.

MySQL: How to hold lock and make other threads wait for an insert that hasn't happened yet

I'm confronted by a seemingly simple problem that has turned out to be pretty hard to figure out. We want to keep a record of every time we're presented with a piece of marketing (a lead) so we don't purchase it more than once in a 90 day period. Many lead providers can present us the same lead many times, often concurrently. We want to return an "accept" to exactly one lead provider.
So let's talk about the scenario that works: We have seen the material in the last 90 days and have a record on the table and there are 3 providers presenting the lead concurrently:
select count(id) from recent_leads where
last_seen_at >= '2019-10-11 00:00:00'
and email = 'yes#example.com' for update;
Thread1 arrives first, and acquires the lock. MySQL returns to Thread1:
+-----------+
| count(id) |
+-----------+
| 1 |
+-----------+
1 row in set (0.00 sec)
Thread1 issues a new insert:
insert into recent_leads (email, last_seen_at)
values ('yes#example.com', '2019-12-12 18:23:35');
Thread2 and Thread3 will blocking trying to execute the same statement until Thread1 commits or issues a rollback on it's transaction. Then Thread2 and Thread3 compete for the lock and the same process happens.
So that works as expected and we're happy with it. The wheels come off when there isn't a record.
Thread1, Thread2, and Thread3 all issue the same SQL as above. MySQL now returns this to all three threads immediately, whereas before, only one Thread would proceed:
+-----------+
| count(id) |
+-----------+
| 0 |
+-----------+
1 row in set (0.00 sec)
All three threads now attempt the insert. Two of them will get an error:
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
Is there a way we can get MySQL to behave like the first scenario all the time? We want Thread2 and Thread3 to block ideally.
Thank you,
-Jonathan
So we ended up dropping locking. Instead, we commit the row in a separate transaction, then select back all rows for an email minus the last_insert_id(). If we find a row with a lower primary key, we assume another thread is already handling the request. This is nice because it's lock free which is a bit easier to debug.

How does GET_LOCK(‘lockname’, 0) of MariaDB/MySQL work?

I have found the use of GET_LOCK(‘lockname’, 0) of MariaDB in a java application that I am working on.
The timeout value is used as 0 here. It should work in non-blocking fashion, I suppose. But, after getting some exceptions in the log file, I have got the impression that it is still trying the get the lock using a default timeout time. Applying the call of IS_FREE_LOCK(‘lockname’) before GET_LOCK call makes the application run smoothly.
My question is, what is the impact of using 0 as the timeout value here?
Have you determined the timeout?. I can't reproduce the problem from the command line:
Session 1:
MariaDB [(none)]> SELECT GET_LOCK('lock1', 10);
+-----------------------+
| GET_LOCK('lock1', 10) |
+-----------------------+
| 1 |
+-----------------------+
1 row in set (0.000 sec)
Session 2:
MariaDB [(none)]> SELECT GET_LOCK('lock1', 0.5);
+------------------------+
| GET_LOCK('lock1', 0.5) |
+------------------------+
| 0 |
+------------------------+
1 row in set (0.500 sec)
MariaDB [(none)]> SELECT GET_LOCK('lock1', 0);
+----------------------+
| GET_LOCK('lock1', 0) |
+----------------------+
| 0 |
+----------------------+
1 row in set (0.000 sec)
"lock wait timeout" has nothing to do with GET_LOCK. It only applies to InnoDB transactions. The default for innodb_lock_wait_timeout is 50 seconds. (In my opinion, that is much too high.)
InnoDB transactions should be designed to finish in very few seconds. Never keep a transaction open while waiting for user interaction; a potty break could lead to "lock wait timeout".
I see that it is a "BatchUpdate". Is this loading a lot of data? Is it coming from some slow source? Could you use autocommit and not put the entire load into a single transaction?
Another thing... If you start a transaction (BEGIN or START TRANSACTION), then do GET_LOCK('foo', 51), and 'foo' is not available, you are asking for "wait lock timeout".
Please provide the bigger picture (BatchUpdate, reason for GET_LOCK, etc) so we can dig deeper.

MySQL InnoDB "SELECT FOR UPDATE" - SKIP LOCKED equivalent

Is there any way to skip "locked rows" when we make "SELECT FOR UPDATE" in MySQL with an InnoDB table?
E.g.: terminal t1
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select id from mytable ORDER BY id ASC limit 5 for update;
+-------+
| id |
+-------+
| 1 |
| 15 |
| 30217 |
| 30218 |
| 30643 |
+-------+
5 rows in set (0.00 sec)
mysql>
At the same time, terminal t2:
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select id from mytable where id>30643 order by id asc limit 2 for update;
+-------+
| id |
+-------+
| 30939 |
| 31211 |
+-------+
2 rows in set (0.01 sec)
mysql> select id from mytable order by id asc limit 5 for update;
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
mysql>
So if I launch a query forcing it to select other rows, it's fine.
But is there a way to skip the locked rows?
I guess this should be a redundant problem in the concurrent process, but I did not find any solution.
EDIT:
In reality, my different concurrent processes are doing something apparently really simple:
take the first rows (which don't contain a specific flag - e.g.: "WHERE myflag_inUse!=1").
Once I get the result of my "select for update", I update the flag and commit the rows.
So I just want to select the rows which are not already locked and where myflag_inUse!=1...
The following link helps me to understand why I get the timeout, but not how to avoid it:
MySQL 'select for update' behaviour
mysql> SHOW VARIABLES LIKE "%version%";
+-------------------------+-------------------------+
| Variable_name | Value |
+-------------------------+-------------------------+
| innodb_version | 5.5.46 |
| protocol_version | 10 |
| slave_type_conversions | |
| version | 5.5.46-0ubuntu0.14.04.2 |
| version_comment | (Ubuntu) |
| version_compile_machine | x86_64 |
| version_compile_os | debian-linux-gnu |
+-------------------------+-------------------------+
7 rows in set (0.00 sec)
MySQL 8.0 introduced support for both SKIP LOCKED and NO WAIT.
SKIP LOCKED is useful for implementing a job queue (a.k.a batch queue) so that you can skip over locks that are already locked by a concurrent transaction.
NO WAIT is useful for avoiding waiting until a concurrent transaction releases the locks that we are also interested in locking.
Without NO WAIT, we either have to wait until the locks are released (at commit or release time by the transaction that currently holds the locks) or the lock acquisition times out. NO WAIT acts as a lock timeout with a value of 0.
For more details about SKIP LOCK and NO WAIT.
This appears to now exist in MySQL starting in 8.0.1:
https://mysqlserverteam.com/mysql-8-0-1-using-skip-locked-and-nowait-to-handle-hot-rows/
Starting with MySQL 8.0.1 we are introducing the SKIP LOCKED modifier
which can be used to non-deterministically read rows from a table
while skipping over the rows which are locked. This can be used by
our booking system to skip orders which are pending. For example:
However, I think that version is not necessarily production ready.
Unfortunately, it seems that there is no way to skip the locked row in a select for update so far.
It would be great if we could use something like the Oracle 'FOR UPDATE SKIP LOCKED'.
In my case, the queries launched in parallel are both exactly the same, and contain a 'where' clause and a 'group by' on a several millions of rows...because the queries need between 20 and 40 seconds to run, that was (as I already knew) a big part of the problem.
The only -temporary and not the best- solution I saw was to move some (i.e.: millions of) rows that I would not (directly) use in order to reduce the time the query will take.
So I will still have the same behavior but I will wait less time...
I was expecting a way to not select the locked row in the select.
I don't mark this as an answer, so if a new clause from mysql is added (or discovered), I can accept it later...
I'm sorry, but I think you approach the problem from a wrong angle. If your user wants to list records from a table that satisfy certain selection criteria, then your query should return them all, or return with an error message and provide no resultset whatsoever. But the query should not reurn only a subset of the results leading the user to belive that he has all the matching records.
The issue should be addressed by making sure that your application locks as few rows as possible, for as little time as possible.
Walk through the table in chunks of the PRIMARY KEY, using some suitable LIMIT so you are not looking at "too many" rows at once.
By using the PK, you are ordering things in a predictable way; this virtually eliminates deadlocks.
By using LIMIT, you will keep from hogging too much at once. The LIMIT should be embodied as a range over the PK. This makes it quite clear if two threads are about to step on each other.
More details are (indirectly) in my blog on big deletes.

MySQL UPDATE operations on InnoDB occasionally timeout

These are simple UPDATEs on very small tables in an InnoDB database. On occasion, an operation appears to lock, and doesn't timeout. Then every subsequent UPDATE ends with a timeout. The only recourse right now is to ask my ISP to restart the daemon. Every field in the table is used in queries, so all the fields are indexed, including a primary.
I'm not sure what causes the initial lock, and my ISP doesn't provide enough information to diagnose the problem. They are reticent about giving me access to any settings as well.
In a previous job, I was required to handle similar information, but instead I would do an INSERT. Periodically, I had a script run to DELETE old records from the table, so that not so many records needed to be filtered. When SELECTing I used extrapolation techniques so having more than just the most recent data was useful. This setup was rock solid, it never hung, even under very heavy usage.
I have no problem replacing the UPDATE with an INSERT and periodic DELETEs, but it just seems so clunky. Has anyone encountered a similar problem and fixed it more elegantly?
Current Configuration
max_heap_table_size: 16 MiB
count(*): 4 (not a typo, four records!)
innodb_buffer_pool_size: 1 GiB
Edit: DB is failing now; locations has 5 records. Sample error below.
MySQL query:
UPDATE locations SET x = "43.630181733", y = "-79.882244160", updated = NULL
WHERE uuid = "6a5c7e9d-400f-c098-68bd-0a0c850b9c86";
MySQL error:
Error #1205 - Lock wait timeout exceeded; try restarting transaction
locations
Field Type Null Default
uuid varchar(36) No
x double Yes NULL
y double Yes NULL
updated timestamp No CURRENT_TIMESTAMP
Indexes:
Keyname Type Cardinality Field
PRIMARY PRIMARY 5 uuid
x INDEX 5 x
y INDEX 5 y
updated INDEX 5 updated
It's a known issue with InnoDB, see MySQL rollback with lost connection. I would welcome something like innodb_rollback_on_disconnect as mentioned there. What's happening to you is that you're getting connections disconnected early, as can happened on the web, and if this happens in the middle of a modifying query, the thread doing that will hang but retain a lock on the table.
Right now, accessing InnoDB directly with web services is vulnerable to these kinds of disconnects and there's nothing you can do within FatCow other than ask them to restart the service for you. Your idea to use MyISAM and low priority is okay, and will probably not have this problem, but if you want to go with InnoDB, would recommend an approach like the following.
1) Go with stored procedures, then the transactions are guaranteed to run to completion and not hang in the event of a disconnect. It's a lot of work, but improves reliability big time.
2) Don't rely on auto commit, ideally set it to zero, and explicitly begin and end each transaction with BEGIN TRANSACTION and COMMIT.
transaction1> START TRANSACTION;
transaction1> SELECT * FROM t WHERE i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 30 |
+------+
transaction2> START TRANSACTION;
transaction2> INSERT INTO t VALUES(26);
transaction2> COMMIT;
transaction1> select * from t where i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 26 |
| 30 |
+------+
What is a gap lock?
A gap lock is a lock on the gap between index records. Thanks to
this gap lock, when you run the same query twice, you get the same
result, regardless other session modifications on that table.
This makes reads consistent and therefore makes the replication
between servers consistent. If you execute SELECT * FROM id > 1000
FOR UPDATE twice, you expect to get the same value twice.
To accomplish that, InnoDB locks all index records found by the
WHERE clause with an exclusive lock and the gaps between them with a
shared gap lock.
This lock doesn’t only affect to SELECT … FOR UPDATE. This is an example with a DELETE statement:
transaction1 > SELECT * FROM t;
+------+
| age |
+------+
| 21 |
| 25 |
| 30 |
+------+
Start a transaction and delete the record 25:
transaction1 > START TRANSACTION;
transaction1 > DELETE FROM t WHERE age=25;
At this point we suppose that only the record 25 is locked. Then, we try to insert another value on the second session:
transaction2 > START TRANSACTION;
transaction2 > INSERT INTO t VALUES(26);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(29);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(23);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(31);
Query OK, 1 row affected (0.00 sec)
After running the delete statement on the first session, not only the affected index record has been locked but also the gap before and after that record with a shared gap lock preventing the insertion of data to other sessions.
If your UPDATE is literally:
UPDATE locations SET updated = NULL;
You are locking all rows in the table. If you abandon the transaction while holding locks on all rows, of course all rows will remain locked. InnoDB is not "unstable" in your environment, it would appear that it is doing exactly what you ask. You need to not abandon the open transaction.