I am using scoped_sessions from SQLAlchemy (to MySQL) and running the SQL commit inside tornado's threadpool. In my unit test, the first time Session.commit() passes but the second Session.commit() hangs. I am closing the session properly after the first time commit. I enabled SQLAlchemy logging and I can see that there is nothing emitted after INSERT INTO ... for the second commit.
Having same issue (hang in later commit) brought me to this question earlier today. I've managed to resolve the issue by making sure that only one session is in place.
In my case I'm running integration tests with session test for each test and doing roll back in tear-down as described in docs. I've got all DB stuff wrapped in DatabaseService class which is responsible for the test setup (a derived TestDatabaseClass doing the session setup in ctor).
The problem was that I managed to initialize the DatabaseService instance (with test setup) twice and experienced the hang in the later created nested session's commit (or in explicit flush call). So making sure that only one DatabaseService exists and all DB queries goes thru one session did resolve the problem for me.
What does you session init look like? Are you not using autoflush? Try calling session.flush() after the commit.
The problem was that I had a pending transaction in between the two tests. That was causing the next commit to just not happen.
Related
I am chasing a problem on a mysql application. At some point my client INSERTs some data, using a query wrapped in a START TRANSACTION; .... COMMIT; statement. Right after that another client comes are read back the data, and it is not there (I am sure of the order of things).
I am running nodejs, express, mysql2, and use connection pooling, with multiple statements queries.
What is interesting is that I see weird things on mysqlworkbench. I just had a workbench instance which would not see the newly inserted data either. I opened a second one, it saw the new data. Minutes later, the first instance would still not see the new data. Hit 'Reconnect to DBMS', and now it sees it. The workbench behaviour, if applied to my node client, would explain the bad result I see in node / mysql2.
There is some sort of caching going on somewhere... no idea where to start :-( Any pointers? Thanks!
It sounds like your clients are living in their own snapshot of the database, which would be true if they have an open transaction using the REPEATABLE-READ isolation level. In other words, no data committed after that client started its transaction will be visible to that client.
One workaround is to force a new transaction to start. Just run COMMIT in the client session where it appears to be viewing stale data. That will resolve any open transaction and the next query will start a new transaction.
Another way you can test is to use a locking read query such as SELECT ... FOR UPDATE. This will read the most recently committed data, regardless of the client's transaction isolation level. That is, even if the client had started their transaction using REPEATABLE-READ, a locking read behaves as if they had started their transaction with READ-COMMITTED.
we're having a weird issue with TypeORM, specifically with Jest(might be related, might not be). A certain test is getting completely stuck/hung and we’re having a hard time figuring out what the issue is.
In terms of stack: Typescript, NodeJS, Apollo Graphql, Jest, MySQL.
The test in question is actually an integration test using Apollo’s integration test framework.
What happened first is that a specific test just completely got stuck, and after several long minutes an error is thrown in the console: QueryFailedError: ER_LOCK_WAIT_TIMEOUT: Lock wait timeout exceeded; try restarting transaction
Trying to pinpoint the problem led me to a function we run on afterEach which “destroys” the database. It initially ran:
await queryRunner.query('DELETE FROM Table1');
await queryRunner.query('DELETE FROM Table2');
...
The error and "deadlock" was initially fixed after I changed it from queryRunner to queryBuilder:
await queryBuilder.delete().from('Table1').execute();
...
This was done after fidgeting around with SHOW PROCESSLIST; and with SHOW ENGINE InnoDB STATUS; to try figuring out what was happening. I also changed the transaction isolation to READ-COMMITTED but to no avail. Nothing really worked except changing it from queryRunner to queryBuilder.
This worked for a bit but now it seems like the test is getting stuck again (The test hasn’t changed but the code it’s testing has). Now after the test hangs, we get this error: Error: Pool is closed. Afterwards the test is "released" and all the tests just start failing one by one.
We found out that this is the sequence of events that causes the test to get stuck:
1. open a transaction with queryRunner
2. perform a read query
3. then perform a write
4. commit the transaction and release the queryRunner
5. delete the DB
6. perform a write - deadlock
Furthermore we noticed the following:
If we make sure that we only use the queryRunner for updates, and not
for queries, then the deadlock doesn’t happen.
Changing the code such that we first make all of the read queries with the regular connection
object (not queryRunner) and only then if we connect with
queryRunner and make all of the writes - then the deadlock does not happen.
Does anyone have any insight as to what might be going on? Is there some instability with queryRunner or some specific things we need to take into account when using it?
Thanks!
I was faced same issue and my problem was unhandled async/await.
Check if some promise object is not handled.
If you use async keyword, you must handle all async functions using await keyword.
Also don't forget calling done method of jest.
I have a long running task that updates some SQLAlchemy objects. A session is opened at the start of the task, updates are made along the way, and the transaction is committed at the end. The problem is that the task runs very long, so the connection will have closed (timed out, "gone away", whatever you want to call it) before the commit is even able to happen. This will cause the commit to fail and the whole task to fail.
This seems absolutely the correct way to do write to a DB to short tasks or non-Celery related things. But it is certainly a problem if the tasks take too long.
Is there some other recommended pattern? Should the Celery task not even utilize the SQLAlchemy objects and instead use some sort of static class whose data can be used to update the actual SQLAlchemy objects, only at the end of the task, maybe? That is the only possible solution I have come up with. I would like to know if there are others or if my idea has other problematic implications.
Open the transaction only when it must occur. Therefore, the session should be closed while the actual task is running, but the objects should be kept around and edited by the task. Then use merge. Make sure the setting expire_on_commit is False.
I have a following setup:
Several data processing workers get configuration from django view get_conf() by http.
Configuration is stored in django model using MySQL / InnoDB backend
Configuration model has overridden save() method which tells workers to reload configuration
I have noticed that sometimes the workers do not receive the changed configuration correctly. In particular, when the conf reload time was shorter than usual, the workers got "old" configuration from get_conf() (missing the most recent change). The transaction model used in Django is the default autocommit.
I have come up with the following possible scenario that could cause the behavior:
New configuration is saved
save() returns but MySQL / InnoDB is still processing the (auto)commit
Workers are booted and make http request for new configuration
MySQL (auto)commit finishes
Is the step 2 in the above scenario possible? That is, can django model save() return before the data is actually committed in the DB if the autocommit transactional method is being used? Or, to go one layer down, can MySQL autocommitting INSERT or UPDATE operation finish before the commit is complete (update / insert visible to other transactions)?
Object may be getting dirty, please try refresh object after save.
obj.save()
obj.refresh_from_db()
reference: https://docs.djangoproject.com/en/1.8/ref/models/instances/#refreshing-objects-from-database
This definitely looks like a race condition.
The scenario you describe should never happen if there's only one script and one database. When you save(), the method doesn't return until the data is actually commited to the database.
If however you're using a master/slave configuration, you could be the victim of the replication delay: if you write on the master but read on the slaves, then it is entirely possible that your script doesn't wait long enough for the replication to occur, and you read the old conf from the slave before it had the opportunity to replicate the master.
Such a configuration can be set up in django using database routers, or it can be done on the DB side by using a DB proxy. Check that out.
On a symfony2 project I'm working on sometimes dead locks occur when calling flush on my entitymanager. This results in an exception. Most of the times this error occures just once and a second attempt to insert the same data is working correctly.
Is there a good approach to execute (flush) the same transaction again. As simple
$em->flush();
won't do, since the entitymanager gets closed if an error occures.
I've found https://github.com/doctrine/doctrine2/pull/806 bit that doesn't provide a solution.
Doctrine throws a RetryableException for this kind of error where you simply have to try it one more time to make it work.
The problem is that with Doctrine 2, those exceptions make the EntityManager unusable and you have to re-instantiate a new one.
Hopefully this will be corrected in Doctrine 3 : issue tracking
Until Doctrine 3 is released, I went with a solution that passed the test of time in production projects in my company. Everything is explained in this blog post
I would use explicit transaction demarcation, to hopefully prevent the deadlock in the first place. By default only flush() is wrapped in a transaction.
Alternatively, you might be able to change your procedure to use the DQL UPDATE query, which should be atomic.
Or re-submit the request back to the action (with some recursion limit).
I'm not sure there is going to be a good way of restarting the entity manager, but keeping the unit of work.