On certain occasions, when several back-end process happen to run at the same time (queue management is something else, I can solve it like that, but this is not the question here),
I get General error: 1205 Lock wait timeout exceeded; try restarting transaction ROLLING BACK
The process which has less priority is the one that locks the table, due to the fact that it started a few minutes before the high priority one.
How do I give priority to a query over an already running process?
Hope it was clear enough.
Once a query has begun execution it cannot be paused/interrupted. The only exception to this is at the DB administration level where you could essentially force the query to stop (think of it as killing a running process in windows if you will). However you don't want to do that, so forget it.
Your best option would be to use a LOW PRIORITY chunked operation. Basically what that means is if the query on the LOW PRIORITY is taking too long to execute, think about ways in which you could split it up to be quicker without creating orphaned data or illegal data in the database.
A very basic use case would be imagine an insert that inserts 10,000 new rows. By "chunking" the insert so that it runs the insert multiple times with smaller data sets (i.e. 500 at a time), each one will complete more quickly, and therefore allow any non-LOW PRIORITY operations to be executed in a more timely manner.
How To
Setting something as low priority is as simple as adding in the LOW_PRIORITY flag.
INSERT LOW_PRIORITY INTO xxx(a,b,c,) VALUES()
UPDATE LOW_PRIORITY xxx SET a=b
DELETE LOW_PRIORITY FROM xxx WHERE a="value"
Related
So basically, I have this current query :
UPDATE act AS a
INNER JOIN blok AS b
ON b.fav_pat = a.pat_id
SET a.blok_id = b.id
Because of the volumn of data i have, its currently timing out. is there a way around to avoid the time out without modifying db config ?
The flyway package you use does its best to allow any incomplete operation to be entirely rolled back using the host RDBMS's transaction semantics. That means it is designed to do update operations like the one you showed us in an ACID-compliant single transaction.
If the tables involved are large (millions of rows or more) the transactions can be very large. They can make your MySQL server thrash, spilling transaction logs to disk or SSD. Committing those transaction logs can take a very long time. You didn't mention row counts, but if they are large is is possible that flyway is not the right tool for this job.
Your lock timeout hints that you are doing this operation on a database with other concurrent activity. You may want to do it on an otherwise quiet database for best results.
You can increase the lock wait timeout by doing this.
show variables like 'innodb_lock_wait_timeout'; -- previous vale
SET GLOBAL innodb_lock_wait_timeout = 300; -- five min
Then, perhaps, try again, just before sunrise on a holiday or another quiet time. More information here.
Consider restoring the lock timeout to its previous value when your flyway job is done.
You can also consider doing your update in batches, for example 1000 rows at a time. But flyway doesn't seem to support that. If you go that route you can ask another question.
Facts:
I have a mysql database with a table name tableA
I am running multiple aws batches at the same time where each batch process communicates with tableA by:
first adding multiple rows to the table
next, deleting multiple rows to the table
Each batch handles its own distinct set of rows
If I run one batch process no problem occurs.
When multiple batch processes run on the same time I got the following error:
sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
It is not related to aws batch, as same problem occurs when I try to do it locally.
Other info:
SELECT ##GLOBAL.transaction_isolation, ##transaction_isolation, ##session.transaction_isolation; ==> repeatable-read, repeatable-read, repeatable-read
show variables like 'innodb_lock_wait_timeout' ==> 50
Question
I can see some solutions recommend to set the innodb_lock_wait_timeout to higher value which will propably eliminate the error. But my understanding is that if I set innodb_lock_wait_timeout to higher value what it will happen is that each transaction will just wait the other transaction to finish. That means that these processes will not run in parallel as each one will wait the other.
What I want is these processes to happen without waiting other transactions(insertions or deletions) that are happening at the moment.
Any recommendations?
Running multiple batch load processes in parallel is difficult.
Speed up the DELETE queries used in your batch process. Run EXPLAIN on them to ensure that they have the indexes they need, then add the indexes you need.
Try using SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; before running your batch in each session. If each batch handles its own distinct set of rows, this may (or may not) allow a bit more parallelism.
Try reducing the size (the row count) of the batches. The performance reason for using transaction batches is to avoid doing a costly COMMIT for every row. You get most of the performance benefit with batches of 100 rows as you do with batches of 10 000 rows.
Try loading each incoming batch into a temporary table outside your transaction. Then use that temporary table inside your transaction to do your update. Something like this code, which is obviously simpler than you need.
CREATE TEMPORARY TABLE batchrows;
INSERT INTO batchrows (col,col,col) VALUES(a,b,c);
INSERT INTO batchrows (col,col,col) VALUES(d,e,f);
INSERT INTO batchrows (col,col,col) VALUES(g,h,i);
BEGIN TRANSACTION;
INSERT INTO maintable SELECT * FROM batchrows;
DELETE FROM maintable WHERE col IN (SELECT whatever FROM batchrows); /* ??? */
COMMIT;
DROP TEMPORARY TABLE batchrows;
The point of this? Reducing the elapsed time during which the transaction lock is held.
Finally: don't try to do batch loading in parallel. Sometimes the integrity of your data simply requires you to process the batches one after another. Actually, that is what happens now in your system: each batch must wait for the previous one to complete.
Generally speaking, Repeatable Read is not a good default for production. It locks all rows it touched. This will create a lot of unnecessary locks. Changing to Read Committed will reduce the locks significantly.
Before other tuning, I suggest you enable innodb locks log to see what are the locks.
set innodb_status_output_locks = on
set innodb_status_output = on
If that lock can be relieved, that will be a big performance boost.
I don't recommend to increase innodb_lock_wait_timeout. If a lock is held more than 50 seconds, the batch job won't be fast.
In a worse scenario which i experienced before, if the database is shared by other application, such as app serer and the long wait timeout could occupy all your connections. This will result your app server cannot serve new requests.
I've a MyISAM table (I can't change it to use InnoDb, do please don't suggest that) which is pretty big (~20GB)
I've a worker which regularly dump this table (I launch is with the --skip-lock-tables option)
During the dump (which takes ~5min), concurrent select can be correctly run, as I would expect. When I go a "REPLACE" during the dump, this REPLACE is "waiting for metadatalock" which seems normal too.
But, every SELECT started after the start the REPLACE will also be "waiting for metadata lock". I can't understand why. Could you help me on this, and tell me how I can have all the selects correctly run (even after this replace)
Thanks !
What is happening is:
Your worker is making a big SELECT. The SELECT is locking the table with a read lock. By the way, the skip-lock-tables only means that you are not locking all the tables at once, but the SELECT query is still locking each table individually. More info on this answer.
Your REPLACE is trying to INSERT but has to wait for the first SELECT (the dump) to finish in order to acquire a write lock. It is put in the write lock queue.
Every SELECT after the REPLACE is put in the read lock queue.
This is a behavior described in the doc on table-level locking:
Table updates are given higher priority than table retrievals. Therefore, when a lock is released, the lock is made available to the requests in the write lock queue and then to the requests in the read lock queue. This ensures that updates to a table are not “starved” even when there is heavy SELECT activity for the table.
If you want the SELECT to not wait for the REPLACE you could (never actually tested that) try the LOW_PRIORITY modifier on your replace.
If you use the LOW_PRIORITY modifier, execution of the INSERT is delayed until no other clients are reading from the table. This includes other clients that began reading while existing clients are reading, and while the INSERT LOW_PRIORITY statement is waiting. It is possible, therefore, for a client that issues an INSERT LOW_PRIORITY statement to wait for a very long time (or even forever) in a read-heavy environment. (This is in contrast to INSERT DELAYED, which lets the client continue at once.)
However be careful as it might never run if there are always a lot of select.
What is the exact difference between the two locking read clauses:
SELECT ... FOR UPDATE
and
SELECT ... LOCK IN SHARE MODE
And why would you need to use one over the other?
I have been trying to understand the difference between the two. I'll document what I have found in hopes it'll be useful to the next person.
Both LOCK IN SHARE MODE and FOR UPDATE ensure no other transaction can update the rows that are selected. The difference between the two is in how they treat locks while reading data.
LOCK IN SHARE MODE does not prevent another transaction from reading the same row that was locked.
FOR UPDATE prevents other locking reads of the same row (non-locking reads can still read that row; LOCK IN SHARE MODE and FOR UPDATE are locking reads).
This matters in cases like updating counters, where you read value in 1 statement and update the value in another. Here using LOCK IN SHARE MODE will allow 2 transactions to read the same initial value. So if the counter was incremented by 1 by both transactions, the ending count might increase only by 1 - since both transactions initially read the same value.
Using FOR UPDATE would have locked the 2nd transaction from reading the value till the first one is done. This will ensure the counter is incremented by 2.
For Update --- You're informing Mysql that the selected rows can be updated in the next steps(before the end of this transaction) ,,so that mysql does'nt grant any read locks on the same set of rows to any other transaction at that moment. The other transaction(whether for read/write )should wait until the first transaction is finished.
For Share- Indicates to Mysql that you're selecting the rows from the table only for reading purpose and not to modify before the end of transaction. Any number of transactions can access read lock on the rows.
Note: There are chances of getting a deadlock if this statement( For update, For share) is not properly used.
Either way the integrity of your data will be guaranteed, it's just a question of how the database guarantees it. Does it do so by raising runtime errors when transactions conflict with each other (i.e. FOR SHARE), or does it do so by serializing any transactions that would conflict with each other (i.e. FOR UPDATE)?
FOR SHARE (a.k.a. LOCK IN SHARE MODE): Transactions face a higher probability of failure due to deadlock, because they delay blocking until the moment an update statement is received (at which point they either block until all readlocks are released, or fail due to deadlock if another write is in progress). However, only one client blocks and eventually succeeds: the other clients will fail with deadlock if they try to update, so only one of them will succeed and the rest will have to retry their transactions.
FOR UPDATE: Transactions won't fail due to deadlock, because they won't be allowed to run concurrently. This may be desirable for example because it makes it easier to reason about multi-threading if all updates are serialized across all clients. However, it limits the concurrency you can achieve because all other transactions block until the first transaction is finished.
Pro-Tip: As an exercise I recommend taking some time to play with a local test database and a couple mysql clients on the command line to prove this behavior for yourself. That is how I eventually understood the difference myself, because it can be very abstract until you see it in action.
I'm developing a mobile application whose backend is developed in Java and database is MySQL.
We have some insert and update operations in database tables with a lot of rows (between 400.000 and 3.000.000). Every operation usually doesn't need to touch every register of the table, but maybe, they are called simultaneously to update a 20% of them.
Sometimes I get this errors:
Deadlock found when trying to get lock; try restarting transaction
and
Lock wait timeout exceeded; try restarting transaction
I have improved my queries making them smaller and faster but I still have a big problem when some operations can't be performed.
My solutions until now have been:
Increase server performance (AWS Instance from m2.large to c3.2xlarge)
SET GLOBAL tx_isolation = 'READ-COMMITTED';
Avoid to check foreign keys: SET FOREIGN_KEY_CHECKS = 0; (I know this is not safe but my priotity is not to lock de database)
Set this values for timeout variables (SHOW VARIABLES LIKE '%timeout%';):
connect_timeout: 10
delayed_insert_timeout: 300
innodb_lock_wait_timeout: 50
innodb_rollback_on_timeout: OFF
interactive_timeout: 28800
lock_wait_timeout: 31536000
net_read_timeout: 30
net_write_timeout: 60
slave_net_timeout: 3600
wait_timeout: 28800
But I'm not sure if these things have decreased performance.
Any idea of how to reduce those errors?
Note: these others SO answer don't help me:
MySQL Lock wait timeout exceeded
MySQL: "lock wait timeout exceeded"
How can I change the default Mysql connection timeout when connecting through python?
Try to update less rows per single transaction.
Instead of updating 20% or rows in a single transaction update 1% of rows 20 times.
This will improve significantly your performances and you will avoid the timeout.
Note: ORM are not the good solution for big updates. It is better to use standard JDBC. Use ORM to retrieve, update, delete few records each time. It speed up the coding phase, not the execution time.
As a comment more than an answer, if you are in the early stages of development, you may wish to consider whether or not you actually need this particular data in a relational database. There are much faster and larger alternatives for storing data from mobile apps depending upon the planned use of the data. [S3 for large files, stored-once, read often (and can be cached); NoSQL (Mongo etc) for unstructured large, write-once, read many, etc.]