I wrote a program to demonstrate the behavior of DTC timeouts with multiple threads. I'm getting several exceptions, seemingly at random. Are all of these simple Timeouts, or are some of them indicative of deeper problems (connection pool interactions, etc)?
The Microsoft Distributed Transaction Coordinator (MS DTC) has cancelled the distributed transaction.
Distributed transaction completed. Either enlist this session in a new transaction or the NULL transaction.
The transaction associated with the current connection has completed but has not been disposed. The transaction must be disposed before the connection can be used to execute SQL statements.
The operation is not valid for the state of the transaction.
ExecuteReader requires an open and available Connection. The connection's current state is closed.
Here's the data part of the code:
using (DemoDataDataContext dc1 = new DemoDataDataContext(Conn1))
using (DemoDataDataContext dc2 = new DemoDataDataContext(Conn2))
{
WriteMany(dc1, 100); //generate 100 records for insert
WriteMany(dc2, 100000); //generate 100,000 records for insert
Console.WriteLine("{0} : {1}", Name, " generated records for insert.");
using (TransactionScope ts = new TransactionScope())
{
dc1.SubmitChanges();
dc2.SubmitChanges();
ts.Complete();
}
}
All these exceptions tell me that you have memory leaks. For me, these exceptions are not just some simple timeouts, I think you should search for deeper problems.
ts.Complete() does not guarantee a commit of the transaction.
It is merely a way of informing the transaction manager of your status. http://msdn.microsoft.com/en-us/library/system.transactions.transactionscope.complete.aspx
Is it an issue that you informed the transaction manager that your transaction is complete when it actually isn't?
Related
I'm trying to understand if the connection_id() function returns the same thread_id for the whole duration of a Transactional annotated method on a Spring application.
I have a method which is annotated with Transactional like so:
#Transactional
#KafkaListener(
topics = "#{kafkaConfiguration.getTopic(T(full.package.to.FlightDaily))}")
public void onMessage(List<ConsumerRecord<String, FlightDaily>> records) {
consume(records, data -> {
final FlightDaily update = data.value();
LOG.log(Level.FINE, "Received Kafka Event, Flight - IataCode: {0}",
update.getFlightNumberIata());
processor.process(update);
});
}
Then, inside the processor, there are a series of operations and multiple entities are inserted / updated on the db.
The table related to one of the entities, has an Audit which is automatically populated by triggers. (AFTER INSERT and AFTER UPDATE inserts)
The trigger grabs the current connection_id with a
DECLARE current_conn_id BIGINT(20);
SET current_conn_id = (SELECT connection_id());
And then insert it as part of the Audit.
The problem is that I was expecting this connection_id to remain stable inside a Transaction.. because from my understanding, it is the thread currently serving my transaction (if the application opens one) but from the Audit content, it seems like it changes a lot and it doesn't remain stable.
I thought it may be that the Spring application is using multiple connections from the pool.. but it doesn't make sense to me that it uses multiple connections of the pool inside the same transaction, shouldn't a single connection remain locked / used by that transaction until it is committed / rolled back?
Update:
The application communicate with a single node of the Galera cluster, until (and if) that node stops working.
I tested my theory with a mysql client, manually starting a transaction and keeping the connection open.. and it seems to remain stable. But of course I have no pool of connections or the transaction automatically handled by the ransactional annotation.
I am using springboot application and connecting to Azure Mysql.
In a method call inside application there is an around spring aspect written which makes call to Azure MySql db.
Following is the squence of queries executed from the aspect and method
#Autowired
EntityManager entityManager
From Aspect : insert into <table name > values ();
For executing this query following piece of code is used
EntityManager newEntityManager = entityManager.getEntityManagerFactory().createEntityManager();
String nativeSql = "insert into table1 values('1','abc;)";
newEntityManager.getTransaction().begin();
try{
newEntityManager.createNativeQuery(nativeSQL).executeUpdate();
entityManager.getTransaction().commit();
} catch (RuntimeException e) {
newEntityManager.getTransaction().rollback();
}finally{
newEntityManager.close();
}
Read calls are done on databse using JPA with annotation
#Transactional(readOnly=true)
Next following piece of executed
EntityManager newEntityManager = entityManager.getEntityManagerFactory().createEntityManager();
String nativeSql = "update table2 set status='dead' where name='abc'";
newEntityManager.getTransaction().begin();
try{
newEntityManager.createNativeQuery(nativeSQL).executeUpdate(); //error occurs at this line
entityManager.getTransaction().commit();
} catch (RuntimeException e) {
newEntityManager.getTransaction().rollback();
}finally{
newEntityManager.close();
}
Following is the complete error
2019-02-10 23:18:00.959 ERROR [bootstrap,c577f32a3b673745,c577f32a3b673745,false] 10628 --- [nio-8106-exec-2] o.h.engine.jdbc.spi.SqlExceptionHelper : Cannot execute statement in a READ ONLY transaction.
But the same code works fine when it the application is connected with local mysql(MariaDb in my case).
This code works fine even if connected with Azure MSSQL.
But the error occurs when connected with Azure MySQL.
Not sure it's the right solution but I had the same problem with MariaDB and it was very confusing. I tried to track the state of the transaction using show variables WHERE Variable_name like 'tx_%' but it always showed tx_read_only as OFF - even when I actually ran read-only transaction. So I don't know how to find out whether the transaction is read-only.
But I pinpointed my problem to a rather obscure scenario. Before the problematic read-write transaction that is reported to be read-only I ran a different read-only transaction. Both used the same physical transaction and "read-only-ness" somehow leaked. But it failed when I used Connection.getMetaData() and then metaData.getColumns(...) in the first RO transaction. We needed to check the columns for a single table.
The problem did not appear, when I switched the transaction reading metadata to read-write, which makes sense if we suspect read-only leaking to another (logical) connection. BTW: we use combination of Hibernate and plain JDBC above a single Hikari connection pool, so this may be factor.
Later we changed the way to find metadata. We prepared a statement with SELECT returning nothing and then asking resultSet.getMetaData(). This was not only much faster for a single table (especially on Oracle, where it took like 5 minutes using Connection.getMetaData()) - but it did not cause the problem, not even when I read the metadata in read-only transaction.
So this was strange - the problem occurred when:
Specific database was used (MariaDB, but MySQL was fine and other types too).
When I read the metadata using Connection.getMetaData() in a read-only transaction (both conditions must be met to assure the failure).
The same wrapped (physical) connection was used for the next read-write transaction.
I checked that the very same JDBC connection was indeed used also in successful scenario when metadata was read from empty SELECT result set using read-only transaction. No problem with the read-write transaction. I thought the problem can be in handling the connection's meta data, I added resultSet.close() to the result set for MetaData.getColumns(), but no help here. In the end I avoided Connection.getMetaData() altogether.
So while I still don't know exactly where the problem is, I could go around it and assure that the connection seems to work for the next read-write transaction indicating no missing cleanup/read-only leak.
Another peculiarity was: We use SET TRANSACTION READ ONLY statement to start read-only transaction for both MySQL and MariaDB. It should work the same way on both databases. When I switched to START TRANSACTION READ ONLY instead, it worked fine. Even when I used the "wrong" way to access table metadata.
But then, sorry, it does not indicate why the OP has problem on MySQL.
We are seeing strange behaviour in our DB transactions- they are not behaving atomically. We use MySQL 5.6.25 Innodb, Eclipselink 2.5.2 as JPA provider and HikariCP 2.6.2 as the connection pool.
This problem surfaces when Eclipselink fails to acquire a connection from the pool during a entityManager.flush call. For sometime, we were swallowing this exception because entry to a particular table was being made on best-effort basis- a sort of audit mode you can say. However,this led to the case where only a part of the transaction was committed- out of 5, only 1,2 or 3 entries were persisted.
To be sure, here are is the flow of events
tx.begin();
em.persist(entity1);
try{
em.persist(entity2);
em.flush(); ---> this is where connection acquisition fails.
} catch(Throwable tx){
//do nothing, except log.
}
em.persist(entity3);
em.flush();
em.persist(entity4);
em.flush();
em.persist(entity5);
em.flush();
em.persist(entity6);
tx.commit();
We are seeing transactions committed till entity3,entity4,entity5, when connection acquisition again fails at some point in the later flushes.
Can anyone point to how exactly this is happening?
The main problem you face is that the connection is not available. Exceptions of that kind must lead to a rollback of the transaction.
Catching those transactions unhandled will change the behaviour of the transaction. The exception during the first em.flush() also obliterates the first em.persist(entity1) which you did not want to lose.
So the solution is to add em.flush() before the try, to make sure, that persisting of entity1 either is guaranteed or leads to an exception which will lead to rollback of the complete transaction.
I would not recommend this kind of solution though.
If persisting of entity2 is optional then normally you can to do that in an extra transaction which means the system will need (for a short time) an additional db-connection for that.
How to create an extra transaction? In Ejb you use the REQUIRES_NEW annotated methods. I am not sure what kind of TransactionManagement you are using here, but I am quite sure that there should be the possibility to create a kind of separate transactions (not to be confused by nested transactions!!).
I have 6 scripts/tasks. Each one of them starts a MySQL transaction, then do its job, which means SELECT/UPDATE/INSERT/DELETE from a MySQL database, then rollback.
So if the database is at a given state S, I launch one task, when the task terminates, the database is back to state S.
When I launch the scripts sequentially, everything works fine:
DB at state S
task 1
DB at state S
task 2
DB at state S
...
...
task 6
DB at state S
But I'd like to speed up the process by multiple-threading and launching the scripts in parallel.
DB at state S
6 tasks at the same time
DB at state S
Some tasks randomly fail, I sometimes get this error:
SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction
I don't understand, I thought transactions were meant for that. Is there something I'm missing ? Any experience, advice, clue is welcome.
The MySQL configuration is:
innodb_lock_wait_timeout = 500
transaction-isolation = SERIALIZABLE
and I add AUTOCOMMIT = 0 at the beginning of each session.
PS: The database was built and used under the REPEATABLE READ isolation level which I changed afterwards.
You can prevent deadlocks by ensuring that every transaction/process does a SELECT...FOR UPDATE on all required data/tables with the same ORDER BY in all cases and with the same order of the tables itself (with at least repeateable read isolation level in MySQL).
Apart from that, isolation levels and transactions are not meant to handle deadlocks, it is vice versa, they are the reason why deadlocks exist. If you encounter a deadlock, there are good chances that you would have an inconsistent state of your dataset (which might be much more serious - if not, you might not need transactions at all).
I'm using the following format for commiting changes to my db using linq.
Begin Transaction (Scope Serialized, Required)
Check Business Rule 1...N
MyDataContext.SubmitChanges()
Save Changes Done In Previous Query To Log File
End Transaction Scope
But in the SQL Server profiler I see the following line in the Connection:Start.
set transaction isolation level read committed
I went through this (http://social.msdn.microsoft.com/Forums/en-US/adodotnetentityframework/thread/93a45026-0425-4d49-a4ac-1b882e90e6d5) and thought I had an answer;
Until I saw this (https://connect.microsoft.com/VisualStudio/feedback/details/565441/transactionscope-linq-to-sql?wa=wsignin1.0) on Microsoft Connect.
Can someone please tell me whether my code is actually executed under Serialized Isolation Level or whether it is infact just running under read committed?
It depends on how you created the transaction.
If you executed inline SQL to begin it (EG. BEGIN TRAN), L2S will not be aware of the transaction and will spin up a new nested one in READ COMMITTED.
However, if you used System.Transaction, or have a transaction set on your DataContext, SubmitChanges will participate in that transaction.
You can see these transaction starting and stopping in Profiler if you choose the TM: Begin Tran and TM: Commit Tran event classes.
Note: ADO.Net does not issue BEGIN TRAN nor does it issue SET TRANSACTION ISOLATION in batches, this is done at a lower level.
If you really want to confirm the behavior, create a trigger on a table that inserts the current isolation level into a logging table and check on it.
You can pick up your current isolation level by running:
SELECT CASE transaction_isolation_level
WHEN 0 THEN 'Unspecified'
WHEN 1 THEN 'Read Uncommitted'
WHEN 2 THEN 'Read Committed'
WHEN 3 THEN 'Repeatable Read'
WHEN 4 THEN 'Serializable'
WHEN 5 THEN 'Snapshot' END AS TRANSACTION_ISOLATION_LEVEL
FROM sys.dm_exec_sessions
where session_id = ##SPID
My guess is you created the DataContext, then used the TransactionScope. You have to open the connection inside the TransactionScope in order for it to enlist.
From http://entityframework.codeplex.com/workitem/1712
TransactionScope uses remote API calls rather than SQL commands to perform transactions in SQL Server. Those API calls are not included in the a standard trace in SQL Profiler.
You can include them by going to the "Event Selection" page, clicking on the "Show all events" checkbox and the selecting all the events from the Transactions category. This will allow you to see when such events as 'TM: Begin Tran Starting', 'SQLTransaction' and 'TM: Begin Tran Completed' actually occur.
You can also check the TransactionID column for the TSQL events in the "Event Selection" page to see the to which transaction each SQL batch being executed is associated.
Unfortunately I don't know of a direct way to observe the effective isolation level under which each command is being executed in SQL Profiler. But there is an indirect way...
When a connection is opened you will see an 'Audit Login' event in the trace. In many cases this event will contain the isolation level. Now, the 'Audit Login' happens before the actual isolation level is set, so the reported isolation level won't accurately reflect the isolation level of the transaction that is about to start. Here are some tips on how to interpret it:
When a connection opening actually hits a new connection it will always report the default transaction isolation level, e.g. you will see 'set transaction isolation level read uncommitted' (as I said, this is unrelated to the effective isolation level of your transaction as that one will be set at a later point)
After a connection has been opened and then returned to the connection pool (i.e. closed), subsequent connection openings will actually reuse that existing connection from the pool. In this case the 'Audit Login' will report the isolation level that was set when the connection got returned to the pool the last time. This can help you see the isolation level that was used, after the fact.
E.g. in your code snippet, the connection is open for a last time to roll back the transaction (because you have not marked the transaction as completed explicitly). In that 'Audit Login' event you should be able to see the isolation level that was effective when the connection was previously used to execute the query, represented by the line 'set transaction isolation level read uncommitted'.