I am using SELECT ... FOR UPDATE statement to achieve row-level locking in my SpringBoot app. Database: MySQL 5.7.28, connector - MariaDb java client 2.5.2, connection pool HikariCP 2.7.9, spring boot version - 2.0.3 Release.
Persistence is accomplished by Spring JDBC Template, not by JPA. I am using Spring Transaction management, annotation-based by slapping #Transactional annotation on my DAO methods. Transactional proxies are generated via AspectJ compile-time weaving (#EnableTransactionManagement(mode = AdviceMode.ASPECTJ)). I am sure that the transaction manager is configured correctly.
I have written a couple of integration tests that would verify the possibility of a race condition when multiple threads are competing to update the same row, that is supposed to be locked with SELECT ... FOR UPDATE.
Now, the tests work 95% of the time, however, there is one test that is failing when there is a particular sequence of ITs executed.
I am certain that when a test fails, a row lock is not imposed.
I have enabled the MySql query log on the server to help with troubleshooting.
Here is what I see when the first thread is executing the SELECT ... FOR UPDATE statement:
2020-01-26T12:54:06.681319Z 1219 Query set autocommit=0
2020-01-26T12:54:36.616097Z 1209 Query SELECT _listed_fields_ FROM _my_table_ WHERE id IN ('19qix6lvsfx') FOR UPDATE
It seems that the auto-commit is set on a wrong connection object. Do I read it right? What are those numbers 1219 and 1209?
When everything works right, the log looks like this:
2020-01-26T13:24:22.940787Z 1243 Query set autocommit=0
2020-01-26T13:24:36.515016Z 1243 Query SELECT _listed_fields_ FROM _my_table_ WHERE id IN ('19xbs7vv53r') FOR UPDATE
Any help will be greatly appreciated.
I dislike autocommit=0 because I am likely to forget to COMMIT eventually.
SELECT ... FOR UPDATE only applies if you are in a transaction (autocommit=0 means you are always in a transaction).
Your snippet of the general log implies that all the connections are setting that value, perhaps repeatedly. No, each connection is unaffected by the setting for other connections.
Do you see any COMMITs in the log? Do you ever issue ROLLBACK?
Related
I am using Spring Boot, Spring Data, JPA(Hibernate) and MySQL in my project. Here is a method signature from my repository interface:
#Lock(LockModeType.PESSIMISTIC_READ)
#QueryHints({ #QueryHint(name = "javax.persistence.lock.timeout", value = "5000") })
#Query("select b from book b where b.id = :bookId")
Optional<Book> findBookCustom(Long bookId);
When I inspect the SQL logs, I can see that "for shared" clause is appended to the select statement.
This is a quote from MySQL documentation regarding shared locks:
If any of these rows were changed by another transaction that has not yet committed, your query waits until that transaction ends and then uses the latest values.
I can actually trigger the blocking behaviour by updating the book from another thread. When I suspend the updating thread (after book is updated) and then start the reading thread, I can see that the reading thread is blocked. The problem is that the reading thread waits for about 50 seconds (this is the default wait value, I suppose) until the timeout exception is thrown. The value 5000 I provide in QueryHint does not take effect. I also tried using the EntityManager directly to issue the JPQL select statement, but the result was same. Any ideas on how i can set the timeout to 5 seconds?
Can we check your MySQL setting via workbench or cmd-line:
select ##innodb_lock_wait_timeout
Then try to set it to a new value by:
set session innodb_lock_wait_timeout = 5
You could now acquire a pessimistic lock through JPA query, and it will set LockTimeout to 5 seconds. For more information check MySQL documentation.
Depending on the persistence provider and database in use, the hint may or may not be observed.
Hope this helps. Cheers!
UPDATE
If one do not wish to modify the session on their MySQL environment. One can put below setting to the application properties file:
spring.jpa.properties.javax.persistence.query.timeout=5000
MySQL does not support query lock timeout and for this reason #QueryHints({ #QueryHint(name = "javax.persistence.lock.timeout", value = "5000") }) is ignored
For more details: https://blog.mimacom.com/handling-pessimistic-locking-jpa-oracle-mysql-postgresql-derbi-h2/
I am using springboot application and connecting to Azure Mysql.
In a method call inside application there is an around spring aspect written which makes call to Azure MySql db.
Following is the squence of queries executed from the aspect and method
#Autowired
EntityManager entityManager
From Aspect : insert into <table name > values ();
For executing this query following piece of code is used
EntityManager newEntityManager = entityManager.getEntityManagerFactory().createEntityManager();
String nativeSql = "insert into table1 values('1','abc;)";
newEntityManager.getTransaction().begin();
try{
newEntityManager.createNativeQuery(nativeSQL).executeUpdate();
entityManager.getTransaction().commit();
} catch (RuntimeException e) {
newEntityManager.getTransaction().rollback();
}finally{
newEntityManager.close();
}
Read calls are done on databse using JPA with annotation
#Transactional(readOnly=true)
Next following piece of executed
EntityManager newEntityManager = entityManager.getEntityManagerFactory().createEntityManager();
String nativeSql = "update table2 set status='dead' where name='abc'";
newEntityManager.getTransaction().begin();
try{
newEntityManager.createNativeQuery(nativeSQL).executeUpdate(); //error occurs at this line
entityManager.getTransaction().commit();
} catch (RuntimeException e) {
newEntityManager.getTransaction().rollback();
}finally{
newEntityManager.close();
}
Following is the complete error
2019-02-10 23:18:00.959 ERROR [bootstrap,c577f32a3b673745,c577f32a3b673745,false] 10628 --- [nio-8106-exec-2] o.h.engine.jdbc.spi.SqlExceptionHelper : Cannot execute statement in a READ ONLY transaction.
But the same code works fine when it the application is connected with local mysql(MariaDb in my case).
This code works fine even if connected with Azure MSSQL.
But the error occurs when connected with Azure MySQL.
Not sure it's the right solution but I had the same problem with MariaDB and it was very confusing. I tried to track the state of the transaction using show variables WHERE Variable_name like 'tx_%' but it always showed tx_read_only as OFF - even when I actually ran read-only transaction. So I don't know how to find out whether the transaction is read-only.
But I pinpointed my problem to a rather obscure scenario. Before the problematic read-write transaction that is reported to be read-only I ran a different read-only transaction. Both used the same physical transaction and "read-only-ness" somehow leaked. But it failed when I used Connection.getMetaData() and then metaData.getColumns(...) in the first RO transaction. We needed to check the columns for a single table.
The problem did not appear, when I switched the transaction reading metadata to read-write, which makes sense if we suspect read-only leaking to another (logical) connection. BTW: we use combination of Hibernate and plain JDBC above a single Hikari connection pool, so this may be factor.
Later we changed the way to find metadata. We prepared a statement with SELECT returning nothing and then asking resultSet.getMetaData(). This was not only much faster for a single table (especially on Oracle, where it took like 5 minutes using Connection.getMetaData()) - but it did not cause the problem, not even when I read the metadata in read-only transaction.
So this was strange - the problem occurred when:
Specific database was used (MariaDB, but MySQL was fine and other types too).
When I read the metadata using Connection.getMetaData() in a read-only transaction (both conditions must be met to assure the failure).
The same wrapped (physical) connection was used for the next read-write transaction.
I checked that the very same JDBC connection was indeed used also in successful scenario when metadata was read from empty SELECT result set using read-only transaction. No problem with the read-write transaction. I thought the problem can be in handling the connection's meta data, I added resultSet.close() to the result set for MetaData.getColumns(), but no help here. In the end I avoided Connection.getMetaData() altogether.
So while I still don't know exactly where the problem is, I could go around it and assure that the connection seems to work for the next read-write transaction indicating no missing cleanup/read-only leak.
Another peculiarity was: We use SET TRANSACTION READ ONLY statement to start read-only transaction for both MySQL and MariaDB. It should work the same way on both databases. When I switched to START TRANSACTION READ ONLY instead, it worked fine. Even when I used the "wrong" way to access table metadata.
But then, sorry, it does not indicate why the OP has problem on MySQL.
I have 6 scripts/tasks. Each one of them starts a MySQL transaction, then do its job, which means SELECT/UPDATE/INSERT/DELETE from a MySQL database, then rollback.
So if the database is at a given state S, I launch one task, when the task terminates, the database is back to state S.
When I launch the scripts sequentially, everything works fine:
DB at state S
task 1
DB at state S
task 2
DB at state S
...
...
task 6
DB at state S
But I'd like to speed up the process by multiple-threading and launching the scripts in parallel.
DB at state S
6 tasks at the same time
DB at state S
Some tasks randomly fail, I sometimes get this error:
SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction
I don't understand, I thought transactions were meant for that. Is there something I'm missing ? Any experience, advice, clue is welcome.
The MySQL configuration is:
innodb_lock_wait_timeout = 500
transaction-isolation = SERIALIZABLE
and I add AUTOCOMMIT = 0 at the beginning of each session.
PS: The database was built and used under the REPEATABLE READ isolation level which I changed afterwards.
You can prevent deadlocks by ensuring that every transaction/process does a SELECT...FOR UPDATE on all required data/tables with the same ORDER BY in all cases and with the same order of the tables itself (with at least repeateable read isolation level in MySQL).
Apart from that, isolation levels and transactions are not meant to handle deadlocks, it is vice versa, they are the reason why deadlocks exist. If you encounter a deadlock, there are good chances that you would have an inconsistent state of your dataset (which might be much more serious - if not, you might not need transactions at all).
We are getting exceptions like this
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#5b7a7896 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#55bc5e2a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#41ca435f (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#460d33b7 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
when load testing our application on MSSQL 2008 R2 (jTDS or official MS JDBC doesn't matter). We never get this exception when running the same tests against PostgreSQL or MySQL.
We don't just want to increase the number of helper threads for c3p0 (which solves the problem, but how long?). We want to know what's the problem as it is workings with other DBMS'.
The applications behaves like:
Send X requests
Wait for a while -> DEADLOCK
Send X requests
Wait for a while -> DEADLOCK
Does anyone know or has an idea why we have this behavior with MSSQL?
Thanks, Adrian
(Btw. BoneCP works without any problem too.)
SQL Server has a much more restrictive locking strategy compared to PostgreSQL or InnoDB.
Especially it will block SELECTs on rows (tables?) that are updated from a different connection/transaction (in the default installation).
You should make sure that you are not selecting the same rows in one session that are being updated from another.
If you can't change the sequence of your code, you might get away with using "dirty reads" in SQL Server.
If I remember that correctly, this is accomplished by adding WITH NOLOCK to the SELECT statements (but I'm not entirely sure)
Edit
A different possibility (if you are on SQL Server 2005 or later) would be to use the new "snapshot isolation" to avoid blocking selects.
In our applications we don't use either ADO.NET transaction or SQL Server transactions in procedures and now we are getting the below error in our website when multiple people are using.
Transaction (Process ID 73) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction
Is this error due to the lack of transactions? I thought the consistency will be handled by the DB itself.
And one thing I noticed that SQLCommand.Timeout property has set to 10000. Will this be an issue for the error?
I am trying to solve this issue ASAP. Please help.
EDIT
I saw the Isolationlevel property of ADO.NET transaction, so if I use ADO.NET transaction with proper isolationlevel property like "ReadUncommitted" during reading and "Serializable" during writing?
Every SQL DML (INSERT, UPDATE, DELETE) or DQL (SELECT) statement runs inside a transaction. The default behaviour for SQL Server is for it to open a new transaction (if one doesn't exist), and if the statement completes without errors, to automatically commit the transaction.
The IMPLICIT_TRANSACTIONS behaviour that Sidharth mentions basically gets SQL Server to change it's behaviour somewhat - it leaves the transaction open when the statement completes.
To get better information in the SQL Server error log, you can turn on a trace flag. This will then tell you which connections were involved in the deadlock (not just the one that got killed), and which resources were involved. You may then be able to determine what pattern of behaviour is leading to the deadlocks.
If you're unable to determine the underlying cause, you may have to add some additional code to your application - that catches sql errors due to deadlocks, and retries the command multiple times. This is usually the last resort - it's better to determine which tables/indexes are involved, and work out a strategy that avoids the deadlocks in the first place.
IsolationLevel is your best bet. Default serialization level of transactions is "Serializable" which is the most stringent and if at this level there is a circular reference chances of deadlock are very high. Set it to ReadCommitted while reading and let it be Serializable while writing.
Sql server can use implicit transactions which is what might be happening in your case. Try setting it off:
SET IMPLICIT_TRANSACTIONS OFF;
Read about it here: http://msdn.microsoft.com/en-us/library/ms190230.aspx