How to fix "too many open files" in MySQL? - mysql

I am very frequently getting this error in MySQL:
OS errno 24 - Too many open files
What's the cause and what are the solutions?

I was getting the errno: 24 - Too many open files too often when i was using many databases at the same time.
Solution
ensure that the connections to db server close propertly
edit /etc/systemd/system.conf. Uncomment and make
DefaultLimitNOFILE=infinity
DefaultLimitMEMLOCK=infinity
then run systemctl daemon-reload and service mysql restart.
You can check the results with the query: SHOW GLOBAL VARIABLES LIKE 'open_files_limit' and you may notice that the value has changed. You should not have any errno 24 now.
Please notice that the solution may differ from other OS/versions. You can try to locate the variables first.Tested with Ubuntu 16.04.3 and mysql 5.7.19.
In my case it was useless to setting up the open_files_limit variable in mysql configuration files as the variable is flagged as a readonly.
I hope it helped!

You probably have a connection leak in your application, that is why open connections are not closed once the function completes it's execution.
I would probably look into the application code and see where the connections/preparedstatement (if it's java) objects are not closed and fix it.
A quick workaround is to increase ulimit of the server (explained here) which would increase number of open file descriptors (i.e. connections). However, if you have a connection leak, you will encounter this error again, at later stages.

I faced the same problem and found a solution on another stackoverflow-question.
By running the following snippet with Bash:
ulimit -n 30000

Related

max_allowed_packet, I don't have MySQL

I'm trying to run sonar-runner.bat, when it almost finished analyzing, it's written max_allowed_packet more than something something. So it fails.
Through deep search, everyone said that i should configer my.ini file inside MySQL folder. But,
I don't have MySQL Installed.
Log:
Error: unable to execute sonar
error: caused by: unable to save file sources
error: caused by:
Error updating database. cause: com.mysql.jdbc.packettoobigexception: packet for query is too large (3215747 >1048576). you can change this..bla..bla
the error may involve org.sonar.core.source.db.filesourcemapper.insert-inline
the error occurred while setting parameters
how can i change it?
help!
As #Fabrice- SonarQube Team suggested that you are running SonarQube Server on top of Mysql. If you want to check you can check within Sonar.properties file.
for removing this issues you have to modify the my.cnf(Linux) or my.ini(for windows).
[mysqld]
max_allowed_packet=256M
if you want to set the same Globally.Log in to Mysql and run the following command.
SET GLOBAL max_allowed_packet=1073741824;
Once you do this settings, Please restart Mysql Server.
I found the answer myself.
Looks like I don't realize how the database works in Sonarqube.
So by DEFAULT, sonarqube use H2. This is a good one, and I believe such my problem won't happened.
Turned out someone from my company actually used his own MYSQL server. So, I found the MySQL folder, change .ini/.cfg file, insert MAX_ALLOWED_PACKET value to bigger number.
VOILA!
Thanks for your help!

Difference In Phpmyadmin Mysql web client and Terminal client

I got problem (#2006 Mysql server gone away) with mysql while connecting and performing some operations through web browser.
Operation Listed below:
When Executing big procedure
Importing database dump
When Access some particular tables It immediately throws "Server gone away".
Refer this question for Scenarios: Record Not Inserted - #2006 Mysql server gone away
Note : The above operations are works fine when I perform through terminal.
I tried some configuration as googing stated. That is set wait_timeout, max_allowed_packet. I checked for the bin_log but it is not available.
But the issues will not rectified.
What is the problem & How can I figure out & fix the issue?
what is the different between access phpmyadmin mysql server from web browser and terminal?
Where I can find the mysql server log file?
Note: If you know about any one of the above questions. Please post here. It would be helpful to trace.
Please help me to figure this out..
Thanks in advance...
Basically nothing except phpMyAdmin is limited by PHP's timeout and resource limits (limits to keep a runaway script from bogging down your entire machine for all eternity; see the docs for details of those values. In some cases, you might be authenticating through a different user account (for instance, root#localhost and root#127.0.0.1 aren't the same user), but as long as you're using a user with the same permissions the differences are minimal.
You can read more about logs in the MySQL manual, note that "By default, no logs are enabled (except the error log on Windows)".
Below are answer for question
From my research the problem is that browser have some limit to disconnect the connection i.e timeout connection. So that the above problem raised.
To resolve this problem
Go to /opt/lampp/phpmyadmin and open config.inc.php
add the command $cfg['ExecTimeLimit'] = 0;
Restart the xamp server. Now you can perform any operations.
`
2. Web client is differ from terminal because Terminal client will not getting timeout. Terminal client maintain the connection till the progress completed. I recommenced to use command prompt to import/export/run process by safe way.
Basically phpmyadmin will not have any log file. If you wanna see warnings and error you should configure the log file.
Configuration steps:
Go to /opt/lampp/etc/my.cnf
Add log_bin = /opt/lampp/var/mysql/filename.log
Restart the xamp server. You can get the log information.

Got an error reading communication packets in MySQL

I am getting the MySQL error
"Got an error reading communication packets"
in MySQL.err file and in my application side I am getting 2013 error (lost connection during query).
All the timeout values are (in seconds):
wait_timeout = 60
net_read_timeout = 30
connect_timeout = 30
How to resolve this?
For what this is worth, this vague error is somewhat common with many possible culprits.
Often it is not a problem with MySql per say...but the system or calling program instead. eg Sometimes you have php memory limits set to low or the swap drive was never setup...that crashes PHP and MySQL is left confused as to what happened.
In my case, I didn't have that, but I was trying to do a json_encode on a character string with non-utf8 characters which would fail silently and drag mysql down with it. On top of this OPCache in PHP7 seems to be seriously buggy and lacks proper error logging. I had to disable that as well...then magically all my problems with "communication packets" in /var/log/mysql/error.log went away.
Hope this helps someone...gave me a LOT of grief.
Another very weird corner case: check your RAM and your swap!
If you have not enough RAM and swap space, your operating system may decide to kill processes causing your exact error in case of MySQL (Got an error reading communication packets). This is a common situation in virtual machines where the provider often does not install any partition for the swap (and when often people does not pay enough RAM).
So, double-check with top or htop to verify if you have ~100% of your RAM in use or if you have not enough swap (or not swap space at all).
In that case, buy more RAM or search "how to install swapfile" in your distribution.
If that's not the case, see the other answers. Cheers! :)

MySQL 5.1 exit code documentation?

Can someone point me at the docs for the termination codes for MySQL 5.1?
I have a MySQL 5.1.41 installation on an Ubuntu server that seems to be crashing about every 90 seconds with the following message in syslog:
init: mysql main precess terminated with status 2
init: mysql main process ended, respawning
Googling for MySQL and "termination code", "termination status", "exit code", or "exit status" yields nothing useful. Apparently MySQL docs call this something different than what init is reporting.
Anyone have a guess?
You'll want to look at the
/var/log/mysql.err
file for more information on why it's crashing. Replace machinename with the hostname of your computer.
UPDATE: I changed the .err file path. It should be correct now. It's based on information found here. It seems to correspond with a file on my Ubuntu installation. Although on my machine, the file is empty.

"MySQL server has gone away" with Ruby on Rails

After our Ruby on Rails application has run for a while, it starts throwing 500s with "MySQL server has gone away". Often this happens overnight. It's started doing this recently, with no obvious change in our server configuration.
Mysql::Error: MySQL server has gone away: SELECT * FROM `widgets`
Restarting the mongrels (not the MySQL server) fixes it.
How can we fix this?
Ruby on Rails 2.3 has a reconnect option for your database connection:
production:
# Your settings
reconnect: true
See:
Ruby on Rails 2.3 Release Notes, sub section 4.8 Reconnecting MySQL Connections.
MySQL auto-reconnect revisited
Good luck!
This is probably caused by the persistent connections to MySQL going away (time out is likely if it's happening over night) and Ruby on Rails is failing to restore the connection, which it should be doing by default:
In the file vendor/rails/actionpack/lib/action_controller/dispatcher.rb is the code:
if defined?(ActiveRecord)
before_dispatch { ActiveRecord::Base.verify_active_connections! }
to_prepare(:activerecord_instantiate_observers) {ActiveRecord::Base.instantiate_observers }
end
The method verify_active_connections! performs several actions, one of which is to recreate any expired connections.
The most likely cause of this error is that this is because a monkey patch has redefined the dispatcher to not call verify_active_connections!, or verify_active_connections! has been changed, etc.
Try ActiveRecord::Base.connection.verify! in Ruby on Rails 4. Verify pings the server and reconnects if it is not connected.
I had this problem when sending really large statements to MySQL. MySQL limits the size of statements and will close the connection if you go over the limit.
set global max_allowed_packet = 1048576; # 2^20 bytes (1 MB) was enough in my case
As the other contributors to this thread have said, it is most likely that MySQL server has closed the connection to your Ruby on Rails application because of inactivity. The default timeout is 28800 seconds, or 8 hours.
set-variable = wait_timeout=86400
Adding this line to your /etc/my.cnf will raise the timeout to 24 hours
http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#option_mysqld_wait_timeout.
Although the documentation doesn't indicate it, a value of 0 may disable the timeout completely, but you would need to experiment as this is just speculation.
There are however three other situations that I know of that can generate that error. The first is the MySQL server being restarted. This will obviously drop all the connections, but as the MySQL client is passive, and this won't be noticed till you do the next query.
The second condition is if someone kills your query from the MySQL command line, and this also drops the connection, because it could leave the client in an undefined state.
The last is if your MySQL server restarts itself due to a fatal internal error. That is, if you are doing a simple query against a table and instantly see 'MySQL has gone away', I'd take a close look at your server's logs to check for hardware error, or database corruption.
First, determine the max_connections in MySQL:
show variables like "max_connections";
You need to make sure that the number of connections you're making in your Ruby on Rails application is less than the maximum allowed number of connections. Note that extra connections can be coming from your cron jobs, delayed_job processes (each would have the same pool size in your database.yml), etc.
Monitor the SQL connections as you go through your application, run processes, etc. by doing the following in MySQL:
show status where variable_name = 'Threads_connected';
You might want to consider closing connections after a Thread finishes execution as database connections do not get closed automatically (I think this is less of an issue with Ruby on Rails 4 applications Reaper):
Thread.new do
begin
# Thread work here
ensure
begin
if (ActiveRecord::Base.connection && ActiveRecord::Base.connection.active?)
ActiveRecord::Base.connection.close
end
rescue
end
end
end
The connection to the MySQL server is probably timing out.
You should be able to increase the timeout in MySQL, but for a proper fix, have your code check that the database connection is still alive, and re-connect if it's not.
Using reconnect: true in the database.yml will cause the database connection to be re-established AFTER the ActiveRecord::StatementInvalid error is raised (As Dave Cheney mentioned).
Unfortunately adding a retry on the database operation seemed necessary to guard against the connection timeout:
begin
do_some_active_record_operation
rescue ActiveRecord::StatementInvalid => e
Rails.logger.debug("Got statement invalid #{e.message} ... trying again")
# Second attempt, now that db connection is re-established
do_some_active_record_operation
end
Do you monitor the number of open MySQL connections or threads? What is your mysql.ini settings for max_connections?
mysql> show status;
Look at Connections, Max_used_connections, Threads_connected, and Threads_created.
You may need to increase the limits in your MySQL configuration, or perhaps rails is not closing the connection properly*.
Note: I've only used Ruby on Rails briefly...
The MySQL documentation for server status is in http://dev.mysql.com/doc/refman/5.0/en/server-status-variables.html.
Something else to check is Unicorn config is correct. See before_fork and after_fork handling of ActiveRecord connection here: https://gist.github.com/nebiros/2776085#file-unicorn-rb
I had this problem in a Ruby on Rails 3 application, using the mysql2 gem. I copied out the offending query and tried running it in MySQL directly, and I got the same error, "MySQL server has gone away.".
The query in question was very, very large. A very large insert (+1 MB). The field I was trying to insert into was a TEXT column and their max size is 64 KB. Rather than throwing an errorm, the connection went away.
I increased the size of the field and got the same thing, so I'm still not sure what the exact issue was. The point is that it was in the database due to some strange query. Anyway!
While forking in Rails.
For anyone running into this while forking in Rails, try clearing the existing connections before forking and then establish a new connection for each fork, like this:
# Clear existing connections before forking to ensure they do not get inherited.
::ActiveRecord::Base.clear_all_connections!
fork do
# Establish a new connection for each fork.
::ActiveRecord::Base.establish_connection
# The rest of the code for each fork...
end
See this StackOverflow answer here: https://stackoverflow.com/a/8915353/293280