I have seen many stack overflows questions and some blogs tried workarounds, but nothings helped - hence re-posting the question with more details.
I am seeing the weird behaviour with MySQL and Python application, details are as follows:
1) My application works perfectly fine with MySQL (tried and tested on many platforms) but on this particular machine it fails to connect to MySQL.
structure of application is :
Windows service -> parent process -> Mysql(child process)
and when application tries to connect to MySQL it get this error:
ERROR 2013 , Lost connection to MySQL server at 'waiting for initial communication packet' - system error 0
I tried:
- connect_timeout=300
- skip-name-resolve=0
- firewall is OFF
- use 17.0.0.1, localhost , IP of machine to connect to but it still fails with same error.
2) Now the weird thing is -
If I manually follow all the steps which application does, It works perfectly fine, details are follows:
a) Start MySQL with same command (which application uses) with administrator privileges
mysql --default-file = xxx --basedir =xxx
b) Connect with same credentials ( -u root -P 6075 -h 127.0.0.1) and
It works perfectly fine, I double checked all the steps which application does, there is no difference between manually steps and application code.
AM I missing something here ? Any suggestions ?
MySQL version : 5.5.35
Python : 2.7
Base OS : Windows 2012 R2
Thanks in advance..
Found a reason - answering my question:
When I used to run MySQL from my application - it was running under system user privileges - so it used to pick "C:\WINDOWS\TEMP" as a temp directory- this directory was messed up - has lot of unnecessary files .. and MySQL was stuck while processing files under this directory...
But when I ran it manually under My administrator account it was using his temp directory... C:\Users\USER_NAME\AppData\Local\Temp and everything was working like magic...
To fix this permanently I changed tmp directory through MySQL conf file and now My Application runs like the Wind.... :)
[mysqld]
tmpdir = 'PATH_TO_THE_DIRECTORY'
I was getting this same error trying to set up a SQL Server Linked Server
Cannot initialize the data source object of OLE DB provider "MSDASQL" for linked server "DBLINKED".
OLE DB provider "MSDASQL" for linked server "DBLINKED" returned message "[MySQL][ODBC 8.0(w) Driver]Lost connection to MySQL server at 'waiting for initial communication packet', system error: 10060". (Microsoft SQL Server, Error: 7303)
You mentioned it in your initial question - the Connection Timeout was the issue for me.
The default is 0 - raised it to 300. I thought default of 0 would mean no timeout, but it's obviously something reasonably short, and I was trying to connect to a remote database on a slow internet connection. A lot of other question and answers out there relate to connecting within the same machine, so this error isn't reported much.
How to force MySQL reconnect at my will in Rails application? I would like to do this either periodically or on DB exceptions like "MySQL server has gone away".
I found ActiveRecord::Base.remove_connection but as it is written, it should be called for some model, not the whole application.
It's a huge pain to restart the Rails console when I'm running it via Heroku with a bunch of objects in variables and then lose my database connection.
The following is code I would not consider "good" to put in your actual application but it temporarily gets over the oft encountered Mysql2::Error: closed MySQL connection in a console:
ActiveRecord::Base.connection.reconnect!
How about using reconnect = true in your database.yml as described here?
Is there any chance to make Rails application connect to old MySQL v4.1.20 server?
rails db
command works fine, but I cannot run the application.
When I generate new app with
rails new app_name -d mysql
I see the following in database.yml file:
MySQL. Versions 4.1 and 5.0 are recommended.
Which should mean I am able to use it.
However Rails Active Record uses MySQL v.5.0.2+ feature (https://github.com/rails/rails/blob/4-0-stable/activerecord/lib/active_record/connection_adapters/abstract_mysql_adapter.rb):
# Make MySQL reject illegal values rather than truncating or blanking them, see
# http://dev.mysql.com/doc/refman/5.0/en/server-sql-mode.html#sqlmode_strict_all_tables
# If the user has provided another value for sql_mode, don't replace it.
if strict_mode? && !variables.has_key?(:sql_mode)
variables[:sql_mode] = 'STRICT_ALL_TABLES'
end
Unfortunately I have nothing to do with old MySQL server version. I have to use it.
Adding
strict: false
to the connector settings in database.yml solved the problem.
I have a problem similar to the one described in Resque is returning Mysql2::Error: closed MySQL connection: SHOW FIELDS FROM `users`.
In my case, I see it during a long-running Rake task (3+ hours). Basically the task runs along importing data from csvs, then suddenly starts spitting "closed MySQL connection" errors. This only happens in our staging env with a remote MySql, can't repro locally.
So my question is, why can the Rails app (using the same connection properties) run for days without issue, but my data import task bombs after a few hours?
Your connections are timing out. This happens all the time on long running rake tasks with the mysql driver. Try setting the reconnect flag in your database.yml
http://guides.rubyonrails.org/2_3_release_notes.html#reconnecting-mysql-connections
Basically just add:
reconnect: true
I am surprised this is not documented more as I run into it all the time. If that doesnt work, try explicitly calling
ActiveRecord::ConnectionAdapters::ConnectionHandler#verify_active_connections!
After our Ruby on Rails application has run for a while, it starts throwing 500s with "MySQL server has gone away". Often this happens overnight. It's started doing this recently, with no obvious change in our server configuration.
Mysql::Error: MySQL server has gone away: SELECT * FROM `widgets`
Restarting the mongrels (not the MySQL server) fixes it.
How can we fix this?
Ruby on Rails 2.3 has a reconnect option for your database connection:
production:
# Your settings
reconnect: true
See:
Ruby on Rails 2.3 Release Notes, sub section 4.8 Reconnecting MySQL Connections.
MySQL auto-reconnect revisited
Good luck!
This is probably caused by the persistent connections to MySQL going away (time out is likely if it's happening over night) and Ruby on Rails is failing to restore the connection, which it should be doing by default:
In the file vendor/rails/actionpack/lib/action_controller/dispatcher.rb is the code:
if defined?(ActiveRecord)
before_dispatch { ActiveRecord::Base.verify_active_connections! }
to_prepare(:activerecord_instantiate_observers) {ActiveRecord::Base.instantiate_observers }
end
The method verify_active_connections! performs several actions, one of which is to recreate any expired connections.
The most likely cause of this error is that this is because a monkey patch has redefined the dispatcher to not call verify_active_connections!, or verify_active_connections! has been changed, etc.
Try ActiveRecord::Base.connection.verify! in Ruby on Rails 4. Verify pings the server and reconnects if it is not connected.
I had this problem when sending really large statements to MySQL. MySQL limits the size of statements and will close the connection if you go over the limit.
set global max_allowed_packet = 1048576; # 2^20 bytes (1 MB) was enough in my case
As the other contributors to this thread have said, it is most likely that MySQL server has closed the connection to your Ruby on Rails application because of inactivity. The default timeout is 28800 seconds, or 8 hours.
set-variable = wait_timeout=86400
Adding this line to your /etc/my.cnf will raise the timeout to 24 hours
http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#option_mysqld_wait_timeout.
Although the documentation doesn't indicate it, a value of 0 may disable the timeout completely, but you would need to experiment as this is just speculation.
There are however three other situations that I know of that can generate that error. The first is the MySQL server being restarted. This will obviously drop all the connections, but as the MySQL client is passive, and this won't be noticed till you do the next query.
The second condition is if someone kills your query from the MySQL command line, and this also drops the connection, because it could leave the client in an undefined state.
The last is if your MySQL server restarts itself due to a fatal internal error. That is, if you are doing a simple query against a table and instantly see 'MySQL has gone away', I'd take a close look at your server's logs to check for hardware error, or database corruption.
First, determine the max_connections in MySQL:
show variables like "max_connections";
You need to make sure that the number of connections you're making in your Ruby on Rails application is less than the maximum allowed number of connections. Note that extra connections can be coming from your cron jobs, delayed_job processes (each would have the same pool size in your database.yml), etc.
Monitor the SQL connections as you go through your application, run processes, etc. by doing the following in MySQL:
show status where variable_name = 'Threads_connected';
You might want to consider closing connections after a Thread finishes execution as database connections do not get closed automatically (I think this is less of an issue with Ruby on Rails 4 applications Reaper):
Thread.new do
begin
# Thread work here
ensure
begin
if (ActiveRecord::Base.connection && ActiveRecord::Base.connection.active?)
ActiveRecord::Base.connection.close
end
rescue
end
end
end
The connection to the MySQL server is probably timing out.
You should be able to increase the timeout in MySQL, but for a proper fix, have your code check that the database connection is still alive, and re-connect if it's not.
Using reconnect: true in the database.yml will cause the database connection to be re-established AFTER the ActiveRecord::StatementInvalid error is raised (As Dave Cheney mentioned).
Unfortunately adding a retry on the database operation seemed necessary to guard against the connection timeout:
begin
do_some_active_record_operation
rescue ActiveRecord::StatementInvalid => e
Rails.logger.debug("Got statement invalid #{e.message} ... trying again")
# Second attempt, now that db connection is re-established
do_some_active_record_operation
end
Do you monitor the number of open MySQL connections or threads? What is your mysql.ini settings for max_connections?
mysql> show status;
Look at Connections, Max_used_connections, Threads_connected, and Threads_created.
You may need to increase the limits in your MySQL configuration, or perhaps rails is not closing the connection properly*.
Note: I've only used Ruby on Rails briefly...
The MySQL documentation for server status is in http://dev.mysql.com/doc/refman/5.0/en/server-status-variables.html.
Something else to check is Unicorn config is correct. See before_fork and after_fork handling of ActiveRecord connection here: https://gist.github.com/nebiros/2776085#file-unicorn-rb
I had this problem in a Ruby on Rails 3 application, using the mysql2 gem. I copied out the offending query and tried running it in MySQL directly, and I got the same error, "MySQL server has gone away.".
The query in question was very, very large. A very large insert (+1 MB). The field I was trying to insert into was a TEXT column and their max size is 64 KB. Rather than throwing an errorm, the connection went away.
I increased the size of the field and got the same thing, so I'm still not sure what the exact issue was. The point is that it was in the database due to some strange query. Anyway!
While forking in Rails.
For anyone running into this while forking in Rails, try clearing the existing connections before forking and then establish a new connection for each fork, like this:
# Clear existing connections before forking to ensure they do not get inherited.
::ActiveRecord::Base.clear_all_connections!
fork do
# Establish a new connection for each fork.
::ActiveRecord::Base.establish_connection
# The rest of the code for each fork...
end
See this StackOverflow answer here: https://stackoverflow.com/a/8915353/293280