We're using Rails v6.0.2.1 on Jruby with managed PostgreSQL (using JDBC adapter) hosted on DigitalOcean. Lately, we have been seeing issues where a certain transaction created some record, threw the Id to Sidekiq for further processing but the record isn't there in the database and Sidekiq fails since the record is not found in db.
There are some Concurrent::Future calls as well inside the transaction. DigitalOcean doesn't show any deadlocks in the said time period. And currently, DO doesn't provide a dump of PostgreSQL logs. How do we debug this?
While looking at Rails production logs, we found out that those transactions didn't log any BEGIN...COMMIT OR ROLLBACK message either.
you might have been using after_save callback on your models I assume? anyway, the issue is that Sidekiq is fast. Sometimes, too fast.
What is happening is that Sidekiq is picking up the job before the transaction is committed. That will trigger the error you are seeing (e.g. RecordNotFound).
You have two solutions: simply retry the job, on a second pass, the record would most likely be in the database again or you can go move to after_commit callback and these errors will disappear.
This issue was discussed here in the past
Related
I am running a multiple spring boot micro-services. All my services connect to database using spring-data(JPA repositories). Unfortunately due to particular API, deadlocks are occurring in database server. As per the primary investigation I feel the deadlock has happened because two transactions trying to modify the same table resource. Should any of it do with isolation levels of transaction.
Yet, I am not able to make concrete conclusion out of it. Can someone please figure out the root cause of this deadlock?
Please see the attached database logs of transactions leading to deadlock in below pictures.
Transaction1
Transaction2
I'm new to MySQL database and got some issue with table lock/deadlock. We are running a system with a heavy transactions run everyday and sometime deadlock happened. I would like to know what happened to the transactions if they exceeded wait timeout. Are they canceled (roll-back) ? Do we need to manually run the transaction again or did application auto retry the transaction after deadlock is resolved?
I'm using MySQL 5.7 with Innodb engine.
Thanks
it dosen't matter what db you are using ,if you are using a transaction it will only be committed on success ie if u look closely there is a commit transaction command at the end of try which u write , unless that line is invoked No changes to the DB will be made hence you can be assured that it will be roll backed at the situation of timeout error
I'm building a background service which boils down to a very complicated queue system. The idea is to use Redis as non-persistent storage, and have a sub/pub scheme which runs on an interval.
All of the subscribers will be behind a load balancer. This removes the complicated problem of maintaining state between all the servers behind the load balancer.
But, this introduces a new problem...how can I ensure that the non-persistent (Redis) and persistent (MySQL) databases are both updated by my application(s)?
It seems like I'm forced to prioritize one, and if I HAVE to prioritize one, I will prioritize persistence. But, in that scenario, what happens if MySQL is updated, Redis is not, and for some reason I have lost the connection to MySQL and cannot undo my last write?
There are two possible solutions to your problem:
Following these steps:
a. Start MySQL transaction with START TRANSACTION
b. Run your MySQL query INSERT INTO ...
c. Run your Redis command
d. Finish your MySQL transaction with COMMIT statement in case if Redis command succeeded or ROLLBACK if command failed
Using transctions ensures that data is consistent in both storages.
Write LUA script for Redis using LuaSQL library (https://realtimelogic.com/ba/doc/en/lua/luasql.html), where you will connect to MySQL, insert your data and then send commands to Redis as well. Then this LUA script can be called from client side with just one command EVAL or EVALSHA
You can try the mysql udf plugin (https://github.com/Ideonella-sakaiensis/lib_mysqludf_redis)
See the post: how to move data from mysql to redis
Ran into this situation recently using SpringBoot (1.2.3) and Flyway (3.1), and could not find much about how to handle:
Server was spinning up and executing a long running alter table add column statement against a mysql database (5.6) 20-30mins. As the script was running the server process was hard terminated since it was not responding to health checks in a given timeframe. Since the MySQL server was processing the statement, it continued to process the statement to completion but the script was not marked as failed or success. When another server was spun up, it tried to execute the script which failed cause the column already existed.
Given that the server could crash at anytime for any reason during a long running script, other than idempotent scripts or a manual db upgrade process, would like to understand established patterns for handling this situation.
Possibly a setting that indicates the server platform uses implicit commits so mark it as run when the script is sent to the server?
You bring up a good point but unfortunately, I don't think Flyway or Spring Boot have any native support for this.
One workaround, ugly as it is, is to implement the beforeEachMigrate and afterEachMigrate callbacks that Flyway provides. You could use them to maintain a separate migration table that keeps track of which migrations have been started and which ones have been completed. Then, if it contains unfinished migrations the next time your application starts, you can shut it down with a descriptive error message.
I recommend creating a feature request about it. If you do, please link us to it!
My approach would be to have separate migration scripts for any long-running SQL that has an implicit commit. Flyway makes it really easy to add minor version numbered scripts, so there's not a good reason to overcomplicate the implementation with what you're suggesting. If you're using PostgreSQL you probably wouldn't need to do this, but Oracle and MySQL would require it.
We recently switched our (ruby) job queueing system from DelayedJob to Resque.
While our latency has gone down, and we've eliminated the database bottleneck, we're now seeing a new problem; one or more of our workers seems to leave a database connection open when it exits. When we look at the process list, there are hundreds of connections in a 'sleep' state. They eventually time out after 90 seconds. We've been throttling back our workers to keep from running out of client connections, but what we really need to find
out is which one (or more) of our jobs is not being polite when it disconnects using the mysql2 ruby client.
Any ideas how we could (1) find the culprits or (2) instrument our code so we can make sure that we are actually disconnecting before the job terminates?
Rails 4.0.x
Resque 1.25.2
mysql2 gem 0.3.16
Make sure your Resque process is disconnecting from the database before forking and re-establishing the connection afterwards. Create an initializer file config/initializers/resque.rb
which contains:
Resque.before_fork do
defined?(ActiveRecord::Base) && ActiveRecord::Base.connection.disconnect!
end
Resque.after_fork do
defined?(ActiveRecord::Base) && ActiveRecord::Base.establish_connection
end
I gave up on Resque and moved to Sidekiq. I'm much happier now.