Ruby - Exception occured: [Mysql2::Error] closed MySQL connection - mysql

Question

I have a Rails application now only runs internally, so there's not so much visits right now. And there're two resque worker running hardly to fetch data from the web and inserts into a mysql database, every insert will followed by sleep 10 second.
We run it on a VPS. After like every 5 hours, I will encounter an Exception Exception occured: [Mysql2::Error] closed MySQL connection".
What could be the reason causing the exception? Now the pool size is 5.
Will it help if I raise the pool size and specify reconnect: true in my database.yml?

Answer 1

This is a common issue when using the mysql2 gem version 0.2.11 or below in combination with multithreading. There is a bug on the issue tracker with a details on the problem but in conclusion the advice is to:
Update the gem version you are using to >= 0.2.12
Add the reconnect: true option your db connection config in the database.yml file
You probably already solved your problem but this might help others who come across this question.

Answer 2

If your workers are inactive for a long period of time, they'll lose their MySQL connection.
see here for the solution
or just stick this in an initializer
unless Rails.env.to_s == 'test'
module ActiveRecord::ConnectionAdapters
class Mysql2Adapter
alias_method :execute_without_retry, :execute
def execute(*args)
execute_without_retry(*args)
rescue Exception => e
if e.message =~ /server has gone away/i
warn "Server timed out, retrying"
reconnect!
retry
else
raise e
end
end
end
end
end

Answer 3

More insight on how to debug this for delayed_job. I did the following after setting reconnect: true on database.yml, and this solution not working.
cd /your_rails_deploy_code/log
cat production.log
# check the pids from delayed job:
E, [2017-02-01T19:45:21.614579 #2592] ERROR -- : 2017-02-01T19:45:21+0000: [Worker(delayed_job.3 host:demeter pid:2592)] Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=193675) FAILED (0 prior attempts) with Mysql2::Error: closed MySQL connection
On my specific case, pid:2592 is the only one constantly failing. Why? Let's find out:
[deploy#demeter] ps -ef | grep 2592
deploy 2592 1 0 Jan31 ? 00:00:40 production/delayed_job.3
deploy 23312 1 0 Feb01 ? 00:00:40 production/delayed_job.1
deploy 23318 1 0 Feb01 ? 00:00:40 production/delayed_job.0
I noticed that specific process started days before my latest deploy. As soon as I killed it, the errors were gone. What I assume happened on my specific case was that my latest deploy did not delete all delayed_job instances correctly.