I am running a multiple spring boot micro-services. All my services connect to database using spring-data(JPA repositories). Unfortunately due to particular API, deadlocks are occurring in database server. As per the primary investigation I feel the deadlock has happened because two transactions trying to modify the same table resource. Should any of it do with isolation levels of transaction.
Yet, I am not able to make concrete conclusion out of it. Can someone please figure out the root cause of this deadlock?
Please see the attached database logs of transactions leading to deadlock in below pictures.
Transaction1
Transaction2
Related
We're using Rails v6.0.2.1 on Jruby with managed PostgreSQL (using JDBC adapter) hosted on DigitalOcean. Lately, we have been seeing issues where a certain transaction created some record, threw the Id to Sidekiq for further processing but the record isn't there in the database and Sidekiq fails since the record is not found in db.
There are some Concurrent::Future calls as well inside the transaction. DigitalOcean doesn't show any deadlocks in the said time period. And currently, DO doesn't provide a dump of PostgreSQL logs. How do we debug this?
While looking at Rails production logs, we found out that those transactions didn't log any BEGIN...COMMIT OR ROLLBACK message either.
you might have been using after_save callback on your models I assume? anyway, the issue is that Sidekiq is fast. Sometimes, too fast.
What is happening is that Sidekiq is picking up the job before the transaction is committed. That will trigger the error you are seeing (e.g. RecordNotFound).
You have two solutions: simply retry the job, on a second pass, the record would most likely be in the database again or you can go move to after_commit callback and these errors will disappear.
This issue was discussed here in the past
Some of the requests to my application fail every day. I want to know every time which all queries went into deadlock.
I am running MySQL 5.6 on AWS.
FYI, tool suggestions are also welcome.
We have a MySQL 5.6 DB server providing service to 10 clients. The clients poll the database server for records to process. We were having intermittent issues where all of the clients would suddenly generate "Lock wait timeout exceeded; try restarting transaction" errors, all of them at the same time. During troubleshooting one of these events we noted that on the server (Windows Server 2008R2) that at the moment the clients generated the error the server time had changed. We took note for the next time.
Today the next time occurred. 9 out of the 10 clients generated the error and sure enough, when we checked the Event Viewer on the server the server time had changed 1 second forward at the exact time the errors were generated.
Can someone explain:
Why this is happening?
Recommend a way to prevent it?
We are already handling deadlock errors and it's actually not clear where in the program this error is coming from. The clients do not poll the server simultaneously but psuedo randomly in approximate 10 second intervals so it's baffling to us that so many would generate the error at the same moment.
Thanks,
Pablo
What may be happening is some thread is holding lock on some record/table (may be an scheduled task is there to take backup of DB which is generating locks) for too long, and your thread is being timed out. As I said in comments please go through innodb status log for more details.
One way to avoid this is if you are using InnoDB storage engine then fine tune your transaction isolation level.
Check transaction isolation level by executing SELECT ##GLOBAL.tx_isolation, ##tx_isolation;
If you see any of these as REPEATABLE READ which is defauld for InnoDB set it to READ COMMITTED using
SET tx_isolation = 'READ-COMMITTED'; and SET GLOBAL tx_isolation = 'READ-COMMITTED';
Firstly this isn't a duplicate of this question because I'm not behind a load balancer at the moment. Also randomly restarting the instance isn't a satisfactory solution either.
I moved our store from using a local mysql db on the EC2 to RDS recently and we're getting some errors on the backend when they're moving categories around / adding products. Mostly they look like SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction but I've seen other variations of PDO errors.
How would one go about eliminating these issues?
I've read suggestions to just 'keep retrying' but this seems stupid.
I've tried tinkering with innodb_lock_wait_timeout and others but doesn't seem to resolve it.
I have finally disabled magento logging to the DB - will see if this helps going forward. Unfortunately I have to wait a while to let the staff tinker on the backend and then see if any fix has an effect.
Thanks
UPDATE
Now also seeing a bunch of these
User Error: DDL statements are not allowed in transactions
I'm using SQLyog to sync a production database to a dev db. On 4 tables, I'm getting:
Error No. 1205 Lock wait timeout exceeded; try restarting transaction
Researching the web seems to indicate that a transaction has begun, locked tables, but has not committed. One post said to SHOW PROCESSLIST; but the only processes appear to be my own, via SQLyog.
I have also tried a Restart of MySQL, but that didn't help either.
As a relative novice in MySQL, I'm stuck: I can't determine what transaction or process is locking the tables, nor how to clear this situation.
Any suggestions would be gratefully accepted!
MTIA
Having the same problem on MySQL-cluster, I've solved (at least it looks being solved now - no fail have occured during last two days) it by performing commit/rollback after SELECTs too.
Export and re-import your database; this can often fix a lot of mysterious problems. You can do this through phpMyAdmin or from the command line.
This page at MediaTemple has a good set of instructions:
http://kb.mediatemple.net/questions/129/Export+and+import+MySQL+databases#gs
(Well, it worked for me!)