I have a Select query which executes on a transactional table having more than 4 million records. Whenever I execute this query , I observe that all write and update operations on that particular transactional table become suspended and we start getting exceptions from java side that lock wait timeout exceeds , try restarting transaction. Currently lock wait timeout is set to 200 seconds. I am unable to understand that why a select statement can create such locks on the table and block all insert/update statements. The table storage engine is InnoDb and primary key is auto-increment key. The MySQL Version is 5.1.40.
Also I m not starting any transaction before executing select statement.
Any Idea?
So, yes, your SELECT in one transaction read-locks the records of that table and write operations which touch the same records will have to wait until read transaction completes (if it follows two phase locking).
This document may help understanding innodb locks model
Related
We have a large table of about 100 million records and 100+ fields and there are frequent select and update queries running related to this table.
Now we have a requirement to set almost 50+ fields to null and we are planning to do this updation based on the primary key.
We are aware that there will be a locking mechanism when two updates are trying to update the same record.
Our question is, what happens when an update and select query is trying to access the same record.
For example in our case
case1: If we are selecting some 10000 records in one thread and during this select query execution if we are trying to update one of this 10000 records to null in another thread, Will both of this query executes without waiting for the other query? how will be the locking mechanism behave in this scenario?
case2: If we are updating 10000 records to null and during this update query execution if we are trying to select one of these 10000 records, Will both of these queries execute without waiting for the other query? how will be the locking mechanism behave in this scenario?
We are using MySQL 5.7, InnoDB engine and consider all parameters in MySQL is default
Apologizing for this basic question
Given your premise that you use InnoDB as the storage engine and default configuration options:
Readers do not block writers, and vice-versa, full stop.
A SELECT query (unless it's a locking read) can read rows even if they are locked by a concurrent transaction. The result of the query will be based on the latest version of the rows that were committed at the time the SELECT query's transaction snapshot started.
Likewise, an UPDATE can lock a row even if it is being read by a concurrent transaction.
Facts:
I have a mysql database with a table name tableA
I am running multiple aws batches at the same time where each batch process communicates with tableA by:
first adding multiple rows to the table
next, deleting multiple rows to the table
Each batch handles its own distinct set of rows
If I run one batch process no problem occurs.
When multiple batch processes run on the same time I got the following error:
sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
It is not related to aws batch, as same problem occurs when I try to do it locally.
Other info:
SELECT ##GLOBAL.transaction_isolation, ##transaction_isolation, ##session.transaction_isolation; ==> repeatable-read, repeatable-read, repeatable-read
show variables like 'innodb_lock_wait_timeout' ==> 50
Question
I can see some solutions recommend to set the innodb_lock_wait_timeout to higher value which will propably eliminate the error. But my understanding is that if I set innodb_lock_wait_timeout to higher value what it will happen is that each transaction will just wait the other transaction to finish. That means that these processes will not run in parallel as each one will wait the other.
What I want is these processes to happen without waiting other transactions(insertions or deletions) that are happening at the moment.
Any recommendations?
Running multiple batch load processes in parallel is difficult.
Speed up the DELETE queries used in your batch process. Run EXPLAIN on them to ensure that they have the indexes they need, then add the indexes you need.
Try using SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; before running your batch in each session. If each batch handles its own distinct set of rows, this may (or may not) allow a bit more parallelism.
Try reducing the size (the row count) of the batches. The performance reason for using transaction batches is to avoid doing a costly COMMIT for every row. You get most of the performance benefit with batches of 100 rows as you do with batches of 10 000 rows.
Try loading each incoming batch into a temporary table outside your transaction. Then use that temporary table inside your transaction to do your update. Something like this code, which is obviously simpler than you need.
CREATE TEMPORARY TABLE batchrows;
INSERT INTO batchrows (col,col,col) VALUES(a,b,c);
INSERT INTO batchrows (col,col,col) VALUES(d,e,f);
INSERT INTO batchrows (col,col,col) VALUES(g,h,i);
BEGIN TRANSACTION;
INSERT INTO maintable SELECT * FROM batchrows;
DELETE FROM maintable WHERE col IN (SELECT whatever FROM batchrows); /* ??? */
COMMIT;
DROP TEMPORARY TABLE batchrows;
The point of this? Reducing the elapsed time during which the transaction lock is held.
Finally: don't try to do batch loading in parallel. Sometimes the integrity of your data simply requires you to process the batches one after another. Actually, that is what happens now in your system: each batch must wait for the previous one to complete.
Generally speaking, Repeatable Read is not a good default for production. It locks all rows it touched. This will create a lot of unnecessary locks. Changing to Read Committed will reduce the locks significantly.
Before other tuning, I suggest you enable innodb locks log to see what are the locks.
set innodb_status_output_locks = on
set innodb_status_output = on
If that lock can be relieved, that will be a big performance boost.
I don't recommend to increase innodb_lock_wait_timeout. If a lock is held more than 50 seconds, the batch job won't be fast.
In a worse scenario which i experienced before, if the database is shared by other application, such as app serer and the long wait timeout could occupy all your connections. This will result your app server cannot serve new requests.
One of my query takes long time(more than 300 seconds) for executing simple query. and fails in statistics state.
It happens while concurrent execution of same query.
"select 1 from <table_name> where id = <value> for update"
Even, i have 'optimizer_search_depth' config as 0 and buffer size has 14GB.
SELECT .... FOR UPDATE sets a read lock on the rows it returns until the transaction is done, therefor when you call the same query multiple times at the same time, the other ones have to wait for the locks to get released
I guess you are using innodb as engine for your table?
Please see innodb-locking-reads for further information on locking with "FOR UPDATE"
In my application (VC++/Windows 8) I am having 7 threads each have opened connection to MySQL database. All these threads concurrently try to increment value of single field in a table.
To do this I've created a sample table DEMO_TABLE having columns MyIndex and MyCounter (both integers) and added a row to it having MyIndex field value 0. Then I am calling executeUpdate through each thread using MySQL Connector C++ :
executeUpdate("UPDATE DEMO_TABLE SET MyCounter = (MyCounter + 1) WHERE MyIndex = 0");
Here I am not using any locking (row or table lock) still the code didn't give me any error or exception. When I checked the value of MyCounter I saw it got increased. So this seems working correct.
But has raised me these questions:
By default MySQL uses MyISAM engine which needs to lock table for concurrent update query execution. But I am not locking table here, how does this code work without throwing any exception?
Does executeUpdate implicitly uses any lock?
(As per my knowledge InnoDB provides row level locking mechanism which I plan to use in my code. But before that I just wanted to try on my own what happens with default engine without any lock. I was expecting I would get some exception which would tell me about race condition so that I can verify the same doesn't happen with the use of lock)
The locking is implicit, yes, but it's not being done by executeUpdate(). The storage engine in MySQL handles the locking and the unlocking.
Any time you write to a MyISAM table, your query waits the write lock on the table to be available, the write lock is acquired, the write is done, and the write lock is released. There is no genuine write concurrency in MyISAM because each worker is in fact waiting in line for the write lock. You don't get an error because the write requests are serialized.
The situation with InnoDB is similar but very different, in that InnoDB only locks a portion of the table, typically at the row level, where InnoDB can lock a range within an index, thereby locking the rows at that range in the index (and the gap that precedes them). This locking is more granular than table locking, allowing improved concurrency behavior, but there is no concurrent operation on the same row -- each worker waits for the lock or locks that it needs.
In both cases, the locks are implicitly taken.
I have a mysql lock question:
If I query this sql: select * from user order by id asc limit 0,1000.
Then anohther thread simutanousely delete the row between 0,1000 in the user table,if allowed?
In the MySQL Documentation for InnoDB, it states InnoDB does locking on the row level and runs queries as nonlocking consistent reads by default.
More directly, however is Internal Locking Methods, which says MySQL uses table-level locking for MyISAM, MEMORY, and MERGE tables, allowing only one session to update those tables at a time. Also, this:
MySQL grants table write locks as follows:
1. If there are no locks on the table, put a write lock on it.
2. Otherwise, put the lock request in the write lock queue.
MySQL grants table read locks as follows:
1. If there are no write locks on the table, put a read lock on it.
2. Otherwise, put the lock request in the read lock queue.
Okay, let's digest that: In InnoDB, each row has it's own lock, which means your query would loop through the table until it hit a row that has a lock. However, in MyISAM, there is only one lock for the entire table, which is set before the query is executed.
In other words, for InnoDB, if the DELETE operation removed the row before the SELECT operation read the row, then the row would not show up in the results. However, if the SELECT operation read the row first, then it would be returned in the result set, but any future SELECT operations would not show the row. If you want to intentionally lock the entire result set in InnoDB, look into SELECT ... FOR UPDATE.
In MyISAM, the table is locked by default, so it depends which query began execution first: if the DELETE operation started first, then the row would not be returned with the SELECT. But if the SELECT operation began execution first, then the row would indeed be returned.
There is more about interlaced here: http://dev.mysql.com/doc/refman/5.0/en/select.html
And also here: Any way to select without causing locking in MySQL?