According to the MySQL documentation, if any session holds a Read lock for a table, then another session requests a Write lock for the same table, the Write lock must be given and the Read lock waits.
I tried it, connected to the MySQL server from two consoles (Windows 7). Locked the table A from the first console (read lock), then tried to lock the same table from the second console (write lock), but the second console just waits till the first lock releases.
Who is wrong: me or the documentation? (MySQL Server version 5.5.27)
The citation from the MySQL official documentation:
"WRITE locks normally have higher priority than READ locks to ensure
that updates are processed as soon as possible. This means that if one
session obtains a READ lock and then another session requests a WRITE
lock, subsequent READ lock requests wait until the session that
requested the WRITE lock has obtained the lock and released it."
It's written right there:
This means that if one session obtains a READ lock and then another session requests a WRITE lock, subsequent READ lock requests wait until the session that requested the WRITE lock has obtained the lock and released it.
The READ locks that were already obtained won't be broken mid-operation. That would cause havoc. It's the subsequent requests that get to wait.
The key word is subsequent in subsequent READ lock requests. I think this is saying existing READ locks will not be paused, but instead, READ locks that occur while a WRITE lock is in effect will be deferred. So I think the docs are right.
Related
We are creating a data pipeline from Mysql in RDS to elastic search for creating search indexes,
and for this using debezium cdc with its mysql source and elastic sink connector.
Now as the mysql is in rds we have to give the mysql user LOCK TABLE permission for two tables we wanted cdc, as mentioned in docs.
We also have various other mysql users performing transactions which may require any of the two tables.
As soon as we connected the mysql connector to our production database there was a lock created and our whole system went down, after realising this we soon stopped the kafka and also removed the connector, but the locks where still increasing and it only solved after we stop all the new queries by stopping our production code from running and manually killing the processes.
What could be the potential cause for this, and how could we prevent this ?
I'm only guessing because I don't know your query traffic. I would assume the locks you saw increasing were the backlog of queries that had been waiting for the table locks to be released.
I mean the following sequence is what I believe happened:
Debezium starts table locks on your two tables.
The application is still working, and it is trying to execute queries that access those locked tables. The queries begin waiting for the lock to be released. They will wait for up to 1 year (this is the default lock_wait_timeout value).
As you spend some minutes trying to figure out why your site is not responding, a large number of blocked queries accumulate. Potentially as many as max_connections. After all the allowed connections are full of blocked queries, then the application cannot connect to MySQL at all.
Finally you stop the Debezium process that is trying to read its initial snapshot of data. It releases its table locks.
Immediately when the table locks are released, the waiting queries can proceed.
But many of them do need to acquire locks too, if they are INSERT/UPDATE/DELETE/REPLACE or if they are SELECT ... FOR UPDATE or other locking statements.
Since there are so many of these queries queued up, it's more likely for them to be requesting locks that overlap, which means they have to wait for each other to finish and release their locks.
Also because there are hundreds of queries executing at the same time, they are overtaxing system resources like CPU, causing high system load, and this makes them all slow down too. So it will take longer for queries to complete, and therefore if they are blocked each other, they have to wait longer.
Meanwhile the application is still trying to accept requests, and therefore is adding more queries to execute. They are also subject to the queueing and resource exhaustion.
Eventually you stop the application, which at least allows the queue of waiting queries to gradually be finished. As the system load goes down, MySQL is able to process the queries more efficiently and finishes them all pretty soon.
The suggestion by the other answer to use a read replica for your Debezium snapshot is a good one. If your application can read from the master MySQL instance for a while, then no query will be blocked on the replica while Debezium has it locked. Eventually Debezium will finish reading all the data, and release the locks, and then go on to read only the binlog. Then the app can resume using the replica as a read instance.
If your binlog uses GTID, you should be able to make a CDC tool like Debezium read the snapshot from the replica, then when that's done, switch to the master to read the binlog. But if you don't use GTID, that's a little more tricky. The tool would have to know the binlog position on the master corresponding to the snapshot on the replica.
If the locking is problem and you cannot afford to tradeoff locking vs consistency then please take a look at snapshot.locking.mode config option.
Use the replica to prevent lock table statement getting executed, why debezium need lock table? all CDC tool fetch the events from bin logs.
The reason is that debezium is not as written in the document (version 1.5). Once FTWRL acquisition fails, it will execute the lock table. It will be released after the snapshot is read. If you see in the log that "Unable to refresh and obtain the global read lock, the table read lock will be used after reading the table name", congratulations, lucky one
I want to use mysql row level lock. I can't lock complete table. I want to avoid two process processing two different message for server at same time.
What I thought that I can have some table called:
server_lock and if one process start working on server it will insert a row in the table.
Problem with this approach is that if application crashes. We need to remove the lock manually.
Is there a way I may row level lock and lock will get released if application is crashing ?
Edit
I am using C++ as language.
My application is similar to message queue. But difference is that there is two queue which are getting populated by one process for each queue. After action if action belong to same object and both are processing same object it may result in wrong data. So I want a locking mechanism b/w these two queue so that both processor don't modify same object at same time.
I can think of two ways:
Implement some error handler on your program where you remove the lock. Without knowing anything about your program it is hard to say how to do this, but most languages have some method to do some work before exiting upon a crash. This is dangerous, because a crash happens when something is not right. If you continue to do any work, it is possible that you corrupt the database or something like that.
Periodically update the lock. Add a thread on your program that periodically reacquires the lock, or reacquire the lock in some loop you are doing. Then, when a lock is not updated in a while, you know that it belonged to a program that crashed.
I have an application with a row that EVERY user reads and updates constantly (almost every request) this causes DB locks and loss of data.
The problem is that it's critical information and everyone must have it synchronized and updated.
I can introduce a small delay in the update info but it must be synchronized for all users.
I'm using Django and MySql.
Note: To keep the application working I'm issuing the update in a separate thread. So even if the update fails it's not waiting for the lock to end to continue.
I am recently reading the InnoDB code and need to write some code on it.
I know in S2PL, a blocked transaction will be resumed after the conflicted transaction finishes. However, I am not sure how InnoDB resumes transactions after a block, is there a thread which handle this kind of work? Thanks a lot.
When a lock is needed but can't be granted, the lock is entered into a lock queue at the page level. When any lock is released, the releasing transaction searches the queue and grants the next newly non-conflicting locks from the queue. See lock_rec_enqueue_waiting, lock_rec_dequeue_from_page, lock_grant, etc. from storage/innobase/lock/lock0lock.c in the MySQL source code.
As per the documentation link given below:
When a lock wait timeout occurs, the current statement is not executed. The current transaction is not rolled back. (Until MySQL 5.0.13 InnoDB rolled back the entire transaction if a lock wait timeout happened. You can restore this behavior by starting the server with the --innodb_rollback_on_timeout option, available as of MySQL 5.0.32.
http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_lock_wait_timeout
Does it mean that when a lock wait timeout occurs, it compromises the transactional integrity?
"roollback on timeout" was the default behaviour till 5.0.13 and I guess that was the correct way to handle such situations. Does anyone think that this should be the default behaviour and the user should not be asked to add a parameter for a functionality that is taken for granted?
It does not compromise referential integrity - it just gives you a chance to either retry, or do something else like commit work completed so far, or rollback.
For small transactions, and for simplicity, you might as well switch on the rollback-on-timeout option. However, if you are running transactions over many hours, you might appreciate the chance to react to a timeout.