MYSQL: Lock wait timeout exceeded; try restarting transaction - mysql

Question

Facts:
I have a mysql database with a table name tableA
I am running multiple aws batches at the same time where each batch process communicates with tableA by:
first adding multiple rows to the table
next, deleting multiple rows to the table
Each batch handles its own distinct set of rows
If I run one batch process no problem occurs.
When multiple batch processes run on the same time I got the following error:
sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
It is not related to aws batch, as same problem occurs when I try to do it locally.
Other info:
SELECT ##GLOBAL.transaction_isolation, ##transaction_isolation, ##session.transaction_isolation; ==> repeatable-read, repeatable-read, repeatable-read
show variables like 'innodb_lock_wait_timeout' ==> 50
Question
I can see some solutions recommend to set the innodb_lock_wait_timeout to higher value which will propably eliminate the error. But my understanding is that if I set innodb_lock_wait_timeout to higher value what it will happen is that each transaction will just wait the other transaction to finish. That means that these processes will not run in parallel as each one will wait the other.
What I want is these processes to happen without waiting other transactions(insertions or deletions) that are happening at the moment.
Any recommendations?

Answer 1

Running multiple batch load processes in parallel is difficult.
Speed up the DELETE queries used in your batch process. Run EXPLAIN on them to ensure that they have the indexes they need, then add the indexes you need.
Try using SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; before running your batch in each session. If each batch handles its own distinct set of rows, this may (or may not) allow a bit more parallelism.
Try reducing the size (the row count) of the batches. The performance reason for using transaction batches is to avoid doing a costly COMMIT for every row. You get most of the performance benefit with batches of 100 rows as you do with batches of 10 000 rows.
Try loading each incoming batch into a temporary table outside your transaction. Then use that temporary table inside your transaction to do your update. Something like this code, which is obviously simpler than you need.
CREATE TEMPORARY TABLE batchrows;
INSERT INTO batchrows (col,col,col) VALUES(a,b,c);
INSERT INTO batchrows (col,col,col) VALUES(d,e,f);
INSERT INTO batchrows (col,col,col) VALUES(g,h,i);
BEGIN TRANSACTION;
INSERT INTO maintable SELECT * FROM batchrows;
DELETE FROM maintable WHERE col IN (SELECT whatever FROM batchrows); /* ??? */
COMMIT;
DROP TEMPORARY TABLE batchrows;
The point of this? Reducing the elapsed time during which the transaction lock is held.
Finally: don't try to do batch loading in parallel. Sometimes the integrity of your data simply requires you to process the batches one after another. Actually, that is what happens now in your system: each batch must wait for the previous one to complete.

Answer 2

Generally speaking, Repeatable Read is not a good default for production. It locks all rows it touched. This will create a lot of unnecessary locks. Changing to Read Committed will reduce the locks significantly.
Before other tuning, I suggest you enable innodb locks log to see what are the locks.
set innodb_status_output_locks = on
set innodb_status_output = on
If that lock can be relieved, that will be a big performance boost.
I don't recommend to increase innodb_lock_wait_timeout. If a lock is held more than 50 seconds, the batch job won't be fast.
In a worse scenario which i experienced before, if the database is shared by other application, such as app serer and the long wait timeout could occupy all your connections. This will result your app server cannot serve new requests.