On Mysql (Amazon RDS), when I try to run the following SQL query
UPDATE
table1 INNER JOIN table2 USING (CommonColumn)
SET
table1.col1 = table2.x,
table1.col2 = table2.y
I get this error after around 52 seconds consistently:
Error Code: 1205. Lock wait timeout exceeded; try restarting transaction
How should I resolve this?
table2 has around 17 million records and table2 which is a subset of table1 has 4 million records. Could it be the size of the tables that is the problem or is something wrong with my query?
Try increasing the default number of IO threads if you are using InnoDB engine. RDS defaults to a measly 4. You will need to restart the DB instance for the Parameter Group to use the new values as they are not "dynamic" (require a reboot).
show variables like 'innodb_%io_threads';
+-------------------------+-------+
| Variable_name | Value |
+-------------------------+-------+
| innodb_read_io_threads | 4 |
| innodb_write_io_threads | 4 |
+-------------------------+-------+
On AWS RDS, I had a DB instance fill up, which caused this error on the client.
I restarted the MySQL instance and the same query worked.
Related
I recently made a copy of my large 32Gig Database to use for my development environment. Now getting this error when I try to create, delete, update items.
SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction
I'm thinking maybe the copy didnt go correctly and some of the tables are locked?
The Database in question is test_dev2 which in use by my dev environment. I tried killing the process for test_dev2 and still same error.
I also restarted the MySQL service and still same error.
I then increased innodb_lock_wait_timeout to 120
+--------------------------+-------+
| Variable_name | Value |
+--------------------------+-------+
| innodb_lock_wait_timeout | 120 |
+--------------------------+-------+
Heres what I get when I run this command SHOW OPEN TABLES WHERE In_use > 0;
+-----------+------------------+--------+-------------+
| Database | Table | In_use | Name_locked |
+-----------+------------------+--------+-------------+
| test_dev2 | core_url_rewrite | 1 | 0 |
+-----------+------------------+--------+-------------+
Any Ideas on how I could fix this 1205 Lock wait timeout exceeded error?
Ask your system Admin if server where you have installed this version has the capability to handle this big DB.
Slim the DB by removing/reducing entries entries
--Remove customers entry, logs entry, quote etc.
Make sure indexing is properly done.
Your Mysql buffer memory size setting too might result in temporary table creation which inturn will slow down execution which in turn will result in time out.
Is there any way to skip "locked rows" when we make "SELECT FOR UPDATE" in MySQL with an InnoDB table?
E.g.: terminal t1
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select id from mytable ORDER BY id ASC limit 5 for update;
+-------+
| id |
+-------+
| 1 |
| 15 |
| 30217 |
| 30218 |
| 30643 |
+-------+
5 rows in set (0.00 sec)
mysql>
At the same time, terminal t2:
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select id from mytable where id>30643 order by id asc limit 2 for update;
+-------+
| id |
+-------+
| 30939 |
| 31211 |
+-------+
2 rows in set (0.01 sec)
mysql> select id from mytable order by id asc limit 5 for update;
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
mysql>
So if I launch a query forcing it to select other rows, it's fine.
But is there a way to skip the locked rows?
I guess this should be a redundant problem in the concurrent process, but I did not find any solution.
EDIT:
In reality, my different concurrent processes are doing something apparently really simple:
take the first rows (which don't contain a specific flag - e.g.: "WHERE myflag_inUse!=1").
Once I get the result of my "select for update", I update the flag and commit the rows.
So I just want to select the rows which are not already locked and where myflag_inUse!=1...
The following link helps me to understand why I get the timeout, but not how to avoid it:
MySQL 'select for update' behaviour
mysql> SHOW VARIABLES LIKE "%version%";
+-------------------------+-------------------------+
| Variable_name | Value |
+-------------------------+-------------------------+
| innodb_version | 5.5.46 |
| protocol_version | 10 |
| slave_type_conversions | |
| version | 5.5.46-0ubuntu0.14.04.2 |
| version_comment | (Ubuntu) |
| version_compile_machine | x86_64 |
| version_compile_os | debian-linux-gnu |
+-------------------------+-------------------------+
7 rows in set (0.00 sec)
MySQL 8.0 introduced support for both SKIP LOCKED and NO WAIT.
SKIP LOCKED is useful for implementing a job queue (a.k.a batch queue) so that you can skip over locks that are already locked by a concurrent transaction.
NO WAIT is useful for avoiding waiting until a concurrent transaction releases the locks that we are also interested in locking.
Without NO WAIT, we either have to wait until the locks are released (at commit or release time by the transaction that currently holds the locks) or the lock acquisition times out. NO WAIT acts as a lock timeout with a value of 0.
For more details about SKIP LOCK and NO WAIT.
This appears to now exist in MySQL starting in 8.0.1:
https://mysqlserverteam.com/mysql-8-0-1-using-skip-locked-and-nowait-to-handle-hot-rows/
Starting with MySQL 8.0.1 we are introducing the SKIP LOCKED modifier
which can be used to non-deterministically read rows from a table
while skipping over the rows which are locked. This can be used by
our booking system to skip orders which are pending. For example:
However, I think that version is not necessarily production ready.
Unfortunately, it seems that there is no way to skip the locked row in a select for update so far.
It would be great if we could use something like the Oracle 'FOR UPDATE SKIP LOCKED'.
In my case, the queries launched in parallel are both exactly the same, and contain a 'where' clause and a 'group by' on a several millions of rows...because the queries need between 20 and 40 seconds to run, that was (as I already knew) a big part of the problem.
The only -temporary and not the best- solution I saw was to move some (i.e.: millions of) rows that I would not (directly) use in order to reduce the time the query will take.
So I will still have the same behavior but I will wait less time...
I was expecting a way to not select the locked row in the select.
I don't mark this as an answer, so if a new clause from mysql is added (or discovered), I can accept it later...
I'm sorry, but I think you approach the problem from a wrong angle. If your user wants to list records from a table that satisfy certain selection criteria, then your query should return them all, or return with an error message and provide no resultset whatsoever. But the query should not reurn only a subset of the results leading the user to belive that he has all the matching records.
The issue should be addressed by making sure that your application locks as few rows as possible, for as little time as possible.
Walk through the table in chunks of the PRIMARY KEY, using some suitable LIMIT so you are not looking at "too many" rows at once.
By using the PK, you are ordering things in a predictable way; this virtually eliminates deadlocks.
By using LIMIT, you will keep from hogging too much at once. The LIMIT should be embodied as a range over the PK. This makes it quite clear if two threads are about to step on each other.
More details are (indirectly) in my blog on big deletes.
These are simple UPDATEs on very small tables in an InnoDB database. On occasion, an operation appears to lock, and doesn't timeout. Then every subsequent UPDATE ends with a timeout. The only recourse right now is to ask my ISP to restart the daemon. Every field in the table is used in queries, so all the fields are indexed, including a primary.
I'm not sure what causes the initial lock, and my ISP doesn't provide enough information to diagnose the problem. They are reticent about giving me access to any settings as well.
In a previous job, I was required to handle similar information, but instead I would do an INSERT. Periodically, I had a script run to DELETE old records from the table, so that not so many records needed to be filtered. When SELECTing I used extrapolation techniques so having more than just the most recent data was useful. This setup was rock solid, it never hung, even under very heavy usage.
I have no problem replacing the UPDATE with an INSERT and periodic DELETEs, but it just seems so clunky. Has anyone encountered a similar problem and fixed it more elegantly?
Current Configuration
max_heap_table_size: 16 MiB
count(*): 4 (not a typo, four records!)
innodb_buffer_pool_size: 1 GiB
Edit: DB is failing now; locations has 5 records. Sample error below.
MySQL query:
UPDATE locations SET x = "43.630181733", y = "-79.882244160", updated = NULL
WHERE uuid = "6a5c7e9d-400f-c098-68bd-0a0c850b9c86";
MySQL error:
Error #1205 - Lock wait timeout exceeded; try restarting transaction
locations
Field Type Null Default
uuid varchar(36) No
x double Yes NULL
y double Yes NULL
updated timestamp No CURRENT_TIMESTAMP
Indexes:
Keyname Type Cardinality Field
PRIMARY PRIMARY 5 uuid
x INDEX 5 x
y INDEX 5 y
updated INDEX 5 updated
It's a known issue with InnoDB, see MySQL rollback with lost connection. I would welcome something like innodb_rollback_on_disconnect as mentioned there. What's happening to you is that you're getting connections disconnected early, as can happened on the web, and if this happens in the middle of a modifying query, the thread doing that will hang but retain a lock on the table.
Right now, accessing InnoDB directly with web services is vulnerable to these kinds of disconnects and there's nothing you can do within FatCow other than ask them to restart the service for you. Your idea to use MyISAM and low priority is okay, and will probably not have this problem, but if you want to go with InnoDB, would recommend an approach like the following.
1) Go with stored procedures, then the transactions are guaranteed to run to completion and not hang in the event of a disconnect. It's a lot of work, but improves reliability big time.
2) Don't rely on auto commit, ideally set it to zero, and explicitly begin and end each transaction with BEGIN TRANSACTION and COMMIT.
transaction1> START TRANSACTION;
transaction1> SELECT * FROM t WHERE i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 30 |
+------+
transaction2> START TRANSACTION;
transaction2> INSERT INTO t VALUES(26);
transaction2> COMMIT;
transaction1> select * from t where i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 26 |
| 30 |
+------+
What is a gap lock?
A gap lock is a lock on the gap between index records. Thanks to
this gap lock, when you run the same query twice, you get the same
result, regardless other session modifications on that table.
This makes reads consistent and therefore makes the replication
between servers consistent. If you execute SELECT * FROM id > 1000
FOR UPDATE twice, you expect to get the same value twice.
To accomplish that, InnoDB locks all index records found by the
WHERE clause with an exclusive lock and the gaps between them with a
shared gap lock.
This lock doesn’t only affect to SELECT … FOR UPDATE. This is an example with a DELETE statement:
transaction1 > SELECT * FROM t;
+------+
| age |
+------+
| 21 |
| 25 |
| 30 |
+------+
Start a transaction and delete the record 25:
transaction1 > START TRANSACTION;
transaction1 > DELETE FROM t WHERE age=25;
At this point we suppose that only the record 25 is locked. Then, we try to insert another value on the second session:
transaction2 > START TRANSACTION;
transaction2 > INSERT INTO t VALUES(26);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(29);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(23);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(31);
Query OK, 1 row affected (0.00 sec)
After running the delete statement on the first session, not only the affected index record has been locked but also the gap before and after that record with a shared gap lock preventing the insertion of data to other sessions.
If your UPDATE is literally:
UPDATE locations SET updated = NULL;
You are locking all rows in the table. If you abandon the transaction while holding locks on all rows, of course all rows will remain locked. InnoDB is not "unstable" in your environment, it would appear that it is doing exactly what you ask. You need to not abandon the open transaction.
I've gotten the next error while trying to perform some bunch deletion with reasonable limit:
query=(DELETE FROM `A` WHERE `id` < 123456 LIMIT 1000)
exception=(1205, 'Lock wait timeout exceeded; try restarting transaction')
And
mysql> SHOW OPEN TABLES like 'A';
+----------+----------------------+--------+-------------+
| Database | Table | In_use | Name_locked |
+----------+----------------------+--------+-------------+
| D | A | 3 | 0 |
+----------+----------------------+--------+-------------+
1 row in set (0.22 sec)
I see that where is might be a deadlock, but show full processlist outputs only itself. Where to dig into?
InnoDB, MySQL 5.5
This means there is a transaction that should be committed. Check other sessions or other applications which may operate with this table.
Also there could be unclosed transacions after SELECTs. I've solved (I hope) such case adding commit/rollback after separate (not some transaciotn parts) SELECTs.
This idea has looked strange for me, so I'd spent some time for other atempts before I've tried it. And it has helped.
I'm building a website with MySQL. I'm using TOAD for MySQL and suddenly I can't connect to the database as I'm getting an error:
"Too many connections"
Is there any way in Toad for MySQL to view existing connections to be able to kill them or simple close all connections all together?
No, there is no built-in MySQL command for that. There are various tools and scripts that support it, you can kill some connections manually or restart the server (but that will be slower).
Use SHOW PROCESSLIST to view all connections, and KILL the process ID's you want to kill.
You could edit the timeout setting to have the MySQL daemon kill the inactive processes itself, or raise the connection count. You can even limit the amount of connections per username, so that if the process keeps misbehaving, the only affected process is the process itself and no other clients on your database get locked out.
If you can't connect yourself anymore to the server, you should know that MySQL always reserves 1 extra connection for a user with the SUPER privilege. Unless your offending process is for some reason using a username with that privilege...
Then after you can access your database again, you should fix the process (website) that's spawning that many connections.
mysql> SHOW PROCESSLIST;
+-----+------+-----------------+------+---------+------+-------+---------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------------+------+---------+------+-------+----------------+
| 143 | root | localhost:61179 | cds | Query | 0 | init | SHOW PROCESSLIST |
| 192 | root | localhost:53793 | cds | Sleep | 4 | | NULL |
+-----+------+-----------------+------+---------+------+-------+----------------+
2 rows in set (0.00 sec)
mysql> KILL 192;
Query OK, 0 rows affected (0.00 sec)
USER 192 :
mysql> SELECT * FROM exept;
+----+
| id |
+----+
| 1 |
+----+
1 row in set (0.00 sec)
mysql> SELECT * FROM exept;
ERROR 2013 (HY000): Lost connection to MySQL server during query
While you can't kill all open connections with a single command, you can create a set of queries to do that for you if there are too many to do by hand.
This example will create a series of KILL <pid>; queries for all some_user's connections from 192.168.1.1 to my_db.
SELECT
CONCAT('KILL ', id, ';')
FROM INFORMATION_SCHEMA.PROCESSLIST
WHERE `User` = 'some_user'
AND `Host` = '192.168.1.1'
AND `db` = 'my_db';
I would recommend checking the connections to show the maximum thread connection is
show variables like "max_connections";
sample
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 13 |
+-----------------+-------+
1 row in set
Then increase it by example
set global max_connections = 500;
In MySQL Workbench:
Left-hand side navigator > Management > Client Connections
It gives you the option to kill queries and connections.
Note: this is not TOAD like the OP asked, but MySQL Workbench users like me may end up here
As above mentioned, there is no special command to do it. However, if all those connection are inactive, using 'flush tables;' is able to release all those connection which are not active.