I've gotten the next error while trying to perform some bunch deletion with reasonable limit:
query=(DELETE FROM `A` WHERE `id` < 123456 LIMIT 1000)
exception=(1205, 'Lock wait timeout exceeded; try restarting transaction')
And
mysql> SHOW OPEN TABLES like 'A';
+----------+----------------------+--------+-------------+
| Database | Table | In_use | Name_locked |
+----------+----------------------+--------+-------------+
| D | A | 3 | 0 |
+----------+----------------------+--------+-------------+
1 row in set (0.22 sec)
I see that where is might be a deadlock, but show full processlist outputs only itself. Where to dig into?
InnoDB, MySQL 5.5
This means there is a transaction that should be committed. Check other sessions or other applications which may operate with this table.
Also there could be unclosed transacions after SELECTs. I've solved (I hope) such case adding commit/rollback after separate (not some transaciotn parts) SELECTs.
This idea has looked strange for me, so I'd spent some time for other atempts before I've tried it. And it has helped.
Related
I seem to not be able extract any helpful information from MySQL to aid my debugging for this error: ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction. Can you help me find some?
Reproduction:
One process does something like this:
start transaction;
update cfgNodes set name="foobar" where ID=29;
and just sits there (not committing, not rolling back). This is clearly the culprit - the process that is hogging the lock because of a long-running transaction - the offender I'm trying to find.
Another process tries:
-- The next line just prevents you from having to wait 50 seconds
set innodb_lock_wait_timeout=1;
update cfgNodes set name="foobar" where ID=29;
This second process gets ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction (after innodb_lock_wait_timeout, default 50secs)
How do I find any information about the culprit?
The standard recommended sources are of little help:
INFORMATION_SCHEMA.INNODB_TRX shows the transaction but not much about it that can help me find it. Only that 1 table is locked (in this bogus tiny example), and that trx_mysql_thread_id is 4093.
mysql> select * from INFORMATION_SCHEMA.INNODB_TRX\G
*************************** 1. row ***************************
trx_id: 280907
trx_state: RUNNING
trx_started: 2018-11-30 00:35:06
trx_requested_lock_id: NULL
trx_wait_started: NULL
trx_weight: 3
trx_mysql_thread_id: 4093
trx_query: NULL
trx_operation_state: NULL
trx_tables_in_use: 0
trx_tables_locked: 1
trx_lock_structs: 2
trx_lock_memory_bytes: 1136
trx_rows_locked: 1
trx_rows_modified: 1
trx_concurrency_tickets: 0
trx_isolation_level: REPEATABLE READ
trx_unique_checks: 1
trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
trx_adaptive_hash_latched: 0
trx_adaptive_hash_timeout: 0
trx_is_read_only: 0
trx_autocommit_non_locking: 0
1 row in set (0.00 sec)
INFORMATION_SCHEMA.INNODB_LOCKS is empty, which makes sense given the documentation, because there is only one transaction and currently nobody waiting for any locks. Also INNODB_LOCKS is deprecated anyway.
SHOW ENGINE INNODB STATUS is useless: cfgNodes is not mentioned at all
SHOW FULL PROCESSLIST is empty, because the culprit is not actually running a query right now.
But now remember the trx_mysql_thread_id from before? We can use that to see the queries executed in that transaction:
mysql> SELECT SQL_TEXT
-> FROM performance_schema.events_statements_history ESH,
-> performance_schema.threads T
-> WHERE ESH.THREAD_ID = T.THREAD_ID
-> AND ESH.SQL_TEXT IS NOT NULL
-> AND T.PROCESSLIST_ID = 4093
-> ORDER BY ESH.EVENT_ID LIMIT 10;
+-----------------------------------------------+
| SQL_TEXT |
+-----------------------------------------------+
| select ##version_comment limit 1 |
| start transaction |
| update cfgNodes set name="foobar" where ID=29 |
+-----------------------------------------------+
3 rows in set (0.00 sec)
Voila - we now see a history of the last 10 queries for each of the remaining transactions allowing us to find the culprit.
I have found the use of GET_LOCK(‘lockname’, 0) of MariaDB in a java application that I am working on.
The timeout value is used as 0 here. It should work in non-blocking fashion, I suppose. But, after getting some exceptions in the log file, I have got the impression that it is still trying the get the lock using a default timeout time. Applying the call of IS_FREE_LOCK(‘lockname’) before GET_LOCK call makes the application run smoothly.
My question is, what is the impact of using 0 as the timeout value here?
Have you determined the timeout?. I can't reproduce the problem from the command line:
Session 1:
MariaDB [(none)]> SELECT GET_LOCK('lock1', 10);
+-----------------------+
| GET_LOCK('lock1', 10) |
+-----------------------+
| 1 |
+-----------------------+
1 row in set (0.000 sec)
Session 2:
MariaDB [(none)]> SELECT GET_LOCK('lock1', 0.5);
+------------------------+
| GET_LOCK('lock1', 0.5) |
+------------------------+
| 0 |
+------------------------+
1 row in set (0.500 sec)
MariaDB [(none)]> SELECT GET_LOCK('lock1', 0);
+----------------------+
| GET_LOCK('lock1', 0) |
+----------------------+
| 0 |
+----------------------+
1 row in set (0.000 sec)
"lock wait timeout" has nothing to do with GET_LOCK. It only applies to InnoDB transactions. The default for innodb_lock_wait_timeout is 50 seconds. (In my opinion, that is much too high.)
InnoDB transactions should be designed to finish in very few seconds. Never keep a transaction open while waiting for user interaction; a potty break could lead to "lock wait timeout".
I see that it is a "BatchUpdate". Is this loading a lot of data? Is it coming from some slow source? Could you use autocommit and not put the entire load into a single transaction?
Another thing... If you start a transaction (BEGIN or START TRANSACTION), then do GET_LOCK('foo', 51), and 'foo' is not available, you are asking for "wait lock timeout".
Please provide the bigger picture (BatchUpdate, reason for GET_LOCK, etc) so we can dig deeper.
I recently made a copy of my large 32Gig Database to use for my development environment. Now getting this error when I try to create, delete, update items.
SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction
I'm thinking maybe the copy didnt go correctly and some of the tables are locked?
The Database in question is test_dev2 which in use by my dev environment. I tried killing the process for test_dev2 and still same error.
I also restarted the MySQL service and still same error.
I then increased innodb_lock_wait_timeout to 120
+--------------------------+-------+
| Variable_name | Value |
+--------------------------+-------+
| innodb_lock_wait_timeout | 120 |
+--------------------------+-------+
Heres what I get when I run this command SHOW OPEN TABLES WHERE In_use > 0;
+-----------+------------------+--------+-------------+
| Database | Table | In_use | Name_locked |
+-----------+------------------+--------+-------------+
| test_dev2 | core_url_rewrite | 1 | 0 |
+-----------+------------------+--------+-------------+
Any Ideas on how I could fix this 1205 Lock wait timeout exceeded error?
Ask your system Admin if server where you have installed this version has the capability to handle this big DB.
Slim the DB by removing/reducing entries entries
--Remove customers entry, logs entry, quote etc.
Make sure indexing is properly done.
Your Mysql buffer memory size setting too might result in temporary table creation which inturn will slow down execution which in turn will result in time out.
Is there any way to skip "locked rows" when we make "SELECT FOR UPDATE" in MySQL with an InnoDB table?
E.g.: terminal t1
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select id from mytable ORDER BY id ASC limit 5 for update;
+-------+
| id |
+-------+
| 1 |
| 15 |
| 30217 |
| 30218 |
| 30643 |
+-------+
5 rows in set (0.00 sec)
mysql>
At the same time, terminal t2:
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select id from mytable where id>30643 order by id asc limit 2 for update;
+-------+
| id |
+-------+
| 30939 |
| 31211 |
+-------+
2 rows in set (0.01 sec)
mysql> select id from mytable order by id asc limit 5 for update;
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
mysql>
So if I launch a query forcing it to select other rows, it's fine.
But is there a way to skip the locked rows?
I guess this should be a redundant problem in the concurrent process, but I did not find any solution.
EDIT:
In reality, my different concurrent processes are doing something apparently really simple:
take the first rows (which don't contain a specific flag - e.g.: "WHERE myflag_inUse!=1").
Once I get the result of my "select for update", I update the flag and commit the rows.
So I just want to select the rows which are not already locked and where myflag_inUse!=1...
The following link helps me to understand why I get the timeout, but not how to avoid it:
MySQL 'select for update' behaviour
mysql> SHOW VARIABLES LIKE "%version%";
+-------------------------+-------------------------+
| Variable_name | Value |
+-------------------------+-------------------------+
| innodb_version | 5.5.46 |
| protocol_version | 10 |
| slave_type_conversions | |
| version | 5.5.46-0ubuntu0.14.04.2 |
| version_comment | (Ubuntu) |
| version_compile_machine | x86_64 |
| version_compile_os | debian-linux-gnu |
+-------------------------+-------------------------+
7 rows in set (0.00 sec)
MySQL 8.0 introduced support for both SKIP LOCKED and NO WAIT.
SKIP LOCKED is useful for implementing a job queue (a.k.a batch queue) so that you can skip over locks that are already locked by a concurrent transaction.
NO WAIT is useful for avoiding waiting until a concurrent transaction releases the locks that we are also interested in locking.
Without NO WAIT, we either have to wait until the locks are released (at commit or release time by the transaction that currently holds the locks) or the lock acquisition times out. NO WAIT acts as a lock timeout with a value of 0.
For more details about SKIP LOCK and NO WAIT.
This appears to now exist in MySQL starting in 8.0.1:
https://mysqlserverteam.com/mysql-8-0-1-using-skip-locked-and-nowait-to-handle-hot-rows/
Starting with MySQL 8.0.1 we are introducing the SKIP LOCKED modifier
which can be used to non-deterministically read rows from a table
while skipping over the rows which are locked. This can be used by
our booking system to skip orders which are pending. For example:
However, I think that version is not necessarily production ready.
Unfortunately, it seems that there is no way to skip the locked row in a select for update so far.
It would be great if we could use something like the Oracle 'FOR UPDATE SKIP LOCKED'.
In my case, the queries launched in parallel are both exactly the same, and contain a 'where' clause and a 'group by' on a several millions of rows...because the queries need between 20 and 40 seconds to run, that was (as I already knew) a big part of the problem.
The only -temporary and not the best- solution I saw was to move some (i.e.: millions of) rows that I would not (directly) use in order to reduce the time the query will take.
So I will still have the same behavior but I will wait less time...
I was expecting a way to not select the locked row in the select.
I don't mark this as an answer, so if a new clause from mysql is added (or discovered), I can accept it later...
I'm sorry, but I think you approach the problem from a wrong angle. If your user wants to list records from a table that satisfy certain selection criteria, then your query should return them all, or return with an error message and provide no resultset whatsoever. But the query should not reurn only a subset of the results leading the user to belive that he has all the matching records.
The issue should be addressed by making sure that your application locks as few rows as possible, for as little time as possible.
Walk through the table in chunks of the PRIMARY KEY, using some suitable LIMIT so you are not looking at "too many" rows at once.
By using the PK, you are ordering things in a predictable way; this virtually eliminates deadlocks.
By using LIMIT, you will keep from hogging too much at once. The LIMIT should be embodied as a range over the PK. This makes it quite clear if two threads are about to step on each other.
More details are (indirectly) in my blog on big deletes.
These are simple UPDATEs on very small tables in an InnoDB database. On occasion, an operation appears to lock, and doesn't timeout. Then every subsequent UPDATE ends with a timeout. The only recourse right now is to ask my ISP to restart the daemon. Every field in the table is used in queries, so all the fields are indexed, including a primary.
I'm not sure what causes the initial lock, and my ISP doesn't provide enough information to diagnose the problem. They are reticent about giving me access to any settings as well.
In a previous job, I was required to handle similar information, but instead I would do an INSERT. Periodically, I had a script run to DELETE old records from the table, so that not so many records needed to be filtered. When SELECTing I used extrapolation techniques so having more than just the most recent data was useful. This setup was rock solid, it never hung, even under very heavy usage.
I have no problem replacing the UPDATE with an INSERT and periodic DELETEs, but it just seems so clunky. Has anyone encountered a similar problem and fixed it more elegantly?
Current Configuration
max_heap_table_size: 16 MiB
count(*): 4 (not a typo, four records!)
innodb_buffer_pool_size: 1 GiB
Edit: DB is failing now; locations has 5 records. Sample error below.
MySQL query:
UPDATE locations SET x = "43.630181733", y = "-79.882244160", updated = NULL
WHERE uuid = "6a5c7e9d-400f-c098-68bd-0a0c850b9c86";
MySQL error:
Error #1205 - Lock wait timeout exceeded; try restarting transaction
locations
Field Type Null Default
uuid varchar(36) No
x double Yes NULL
y double Yes NULL
updated timestamp No CURRENT_TIMESTAMP
Indexes:
Keyname Type Cardinality Field
PRIMARY PRIMARY 5 uuid
x INDEX 5 x
y INDEX 5 y
updated INDEX 5 updated
It's a known issue with InnoDB, see MySQL rollback with lost connection. I would welcome something like innodb_rollback_on_disconnect as mentioned there. What's happening to you is that you're getting connections disconnected early, as can happened on the web, and if this happens in the middle of a modifying query, the thread doing that will hang but retain a lock on the table.
Right now, accessing InnoDB directly with web services is vulnerable to these kinds of disconnects and there's nothing you can do within FatCow other than ask them to restart the service for you. Your idea to use MyISAM and low priority is okay, and will probably not have this problem, but if you want to go with InnoDB, would recommend an approach like the following.
1) Go with stored procedures, then the transactions are guaranteed to run to completion and not hang in the event of a disconnect. It's a lot of work, but improves reliability big time.
2) Don't rely on auto commit, ideally set it to zero, and explicitly begin and end each transaction with BEGIN TRANSACTION and COMMIT.
transaction1> START TRANSACTION;
transaction1> SELECT * FROM t WHERE i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 30 |
+------+
transaction2> START TRANSACTION;
transaction2> INSERT INTO t VALUES(26);
transaction2> COMMIT;
transaction1> select * from t where i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 26 |
| 30 |
+------+
What is a gap lock?
A gap lock is a lock on the gap between index records. Thanks to
this gap lock, when you run the same query twice, you get the same
result, regardless other session modifications on that table.
This makes reads consistent and therefore makes the replication
between servers consistent. If you execute SELECT * FROM id > 1000
FOR UPDATE twice, you expect to get the same value twice.
To accomplish that, InnoDB locks all index records found by the
WHERE clause with an exclusive lock and the gaps between them with a
shared gap lock.
This lock doesn’t only affect to SELECT … FOR UPDATE. This is an example with a DELETE statement:
transaction1 > SELECT * FROM t;
+------+
| age |
+------+
| 21 |
| 25 |
| 30 |
+------+
Start a transaction and delete the record 25:
transaction1 > START TRANSACTION;
transaction1 > DELETE FROM t WHERE age=25;
At this point we suppose that only the record 25 is locked. Then, we try to insert another value on the second session:
transaction2 > START TRANSACTION;
transaction2 > INSERT INTO t VALUES(26);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(29);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(23);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(31);
Query OK, 1 row affected (0.00 sec)
After running the delete statement on the first session, not only the affected index record has been locked but also the gap before and after that record with a shared gap lock preventing the insertion of data to other sessions.
If your UPDATE is literally:
UPDATE locations SET updated = NULL;
You are locking all rows in the table. If you abandon the transaction while holding locks on all rows, of course all rows will remain locked. InnoDB is not "unstable" in your environment, it would appear that it is doing exactly what you ask. You need to not abandon the open transaction.