I have a MySQL server that is hosting my website. Yesterday, after trying to empty a certain database, I started receiving the following error when attempting to connect to my site :
PDOException: SQLSTATE[HY000] [1040] Too many connections in lock_may_be_available()
(line 167 of /home/vksappc/public_html/prod/includes/lock.inc).
After searching online for a possible solution, I restarted the server and everything was okay. Today, however, I've noticed that the number of active connections is constantly and slowly rising until it reaches the limit of 151 :
Screenshot of thread count
At which point the server then seems to timeout those connections and everything resets, only to start its gradual climb all over again:
Screenshot of new thread count
This is really becoming a big problem and as I am quite new to managing a server, I am not too sure what could be causing this or what to do next.
Any ideas?
Turns out there was a specific connection that was stuck in a loop of sending a request unsuccessfully and trying over and over again. Once we found which connection it was, we were able to kill all its processes and fix the issue. Thanks to #WilsonHauck for the tip in the comments.
Related
Error displaying the error page: Application Instantiation Error: Could not connect to MySQL.
This Error is shown sometimes (one every 20 pageviews) and I tried a lot of solutions without any change.
This error found with Joomla 3.5+
This will help you do some testings...
On MySql server run this query...
show processlist;
If this shows a lot of connections then you may be hitting the max_connections problem. Try to increase that to a reasonable number. But, do not make it too much because if it goes beyond the amount of available resources on your server then it will make MySql server crash and it will become a bigger problem where your website will go offline.
Try to check if you are closing sessions properly in your scripts using session_write_close(); You can find more about this function here: http://php.net/manual/en/function.session-write-close.php
This function will write user's current session on disk to be reloaded next time and most important part it will do for you with this problem is that it will close PHP's any connection made to the MySql server, so it will free up the MySql server for next visitor to come and open another page.
Lets say if you have 150 max_connections and at the same time you get 200 visitors, then PHP will still wait for some time for another 50 connections to be free before giving a timeout error where you get "Could not connect to server". But, if sessions are not closing properly, then its so bad that 50 connections will never get processed because 150 will never go away very soon, there is a separate MySql setting for that named wait_timeout. So, any MySql connection you open will keep waiting until you close yourself in script just like you open OR until wait_timeout happens. But, if you close sessions properly then next connection will become available as soon as your PHP script has finished processing and page is generated for the user for viewing and it will make the connection seat free for next connection to come in.
Server RAM was very low!
After upgrade from 0.5GB to 2GB all the kind of error starts with Error displaying the error page: gone.
I found the problem solution by using htop while connecting to the server via SSH, after that, I start opening pages rapidly which show the error plus the htop shows that the RAM was close to full.
show processlist; were helpful to find that the problem not about the connection number.which it may sometimes cost the same error in the same way.
Yesterday, my website crashed. Or, what appears to be a crash.
My host (JustHost) have disclosed that it was a SAN problem and have been rebuilding it. It's hard to get anymore information out of them because they are playing their cards very close to their chest for some reason.
I have suffered almost 24 hours downtime in 2 days and I'm getting fed up.
The error I am receiving on the frontend of the website is:
Database connection error (2): Could not connect to MySQL.Database connection error (2): Could not connect to MySQL.
Trying to access my phpMyadmin from cPanel gives me the following error:
Accessing SSH and trying to start the mysql service via putty I get:
Starting MySQL. ERROR! The server quit without updating PID file (/var/lib/mysql /mysqld.pid).
Can anyone please help me resolve this issue as my host is rather useless!
Thanks!
Yep, mysqld is down. Did you try running "mysqld_safe&"?
For anyone who wishes to know the conclusion of this issue.
I have been left with 2 choices:
1) Be patient and wait for Just Host to fix the issue and potentially lose my Google search rankings and a lot more future revenue
2) Set up a new hosting account, transfer my data over and wait for the domain to propagate.
I have chosen the latter of the two. My confidence in Just Host as a customer service hub and web hosting provider has thoroughly diminished to the point where all of my hosting accounts will now be switched over to another provider.
They have let me down by not fixing the issue in good time and have passed me from pillar to post in live chat, email support and phone support. The final nail in the coffin was my request for them to backup my databases and send them to me via email being met with the resoundingly passive "sorry, we do not provide that service" response.
I would do it myself if only I had access to phpMyadmin, which I do not have because of a server error that they have been incapable of fixing for the past 72 hours.
Goodbye Just Host.
I'm using Node.js to run a web-server for my web application. I'm also using the node-mysql module to interface with a MySQL server for all my persistent database needs.
Whenever there is a critical error within my Node.js application that crashes my app's process I get an email sent to me. So, I keep getting this email with an error saying "Too many connections". Here's an example of the error:
Error: Too many connections
at Function.Client._packetToUserObject (/apps/x/node_modules/mysql/lib/client.js:394:11)
at Client._handlePacket (/apps/x/node_modules/mysql/lib/client.js:307:43)
at Parser.EventEmitter.emit (events.js:96:17)
at Parser.write.emitPacket (/apps/x/node_modules/mysql/lib/parser.js:71:14)
at Parser.write (/apps/x/node_modules/mysql/lib/parser.js:576:7)
at Socket.EventEmitter.emit (events.js:96:17)
at TCP.onread (net.js:396:14)
As you can see all it tells me is that the error is coming from the mysql module, but it doesn't tell me where in my application code the issue is occurring.
My application opens a db connection anytime I need to run one or more queries. I immediately close the connection after all my queries and data has been collected. So, I don't understand how I could be exceeding the 151 max_connections limit.
Unless there is a place in my code where I forgot to call db.end() to close the connection, I don't see how my app would leak like this. Even if there was such a mistake, I wouldn't get these emails sent by the dozens. Yesterday, I received almost 100 emails with roughly the same error. How could this be happening? If my application had leaked and allocated connections over time, as soon as the first error occurred the app process would crash and all connections would be lost, preventing the app to crash again. Since I received ~100 emails, this means the app crashed ~100 times, and all within a short period of time. This could only mean that somewhere in my application a lot of connections where established in a short period of time, right?
How could I avoid this problem? This is very discouraging. All help is highly appreciated. Thanks
MySQL has a default MAX_CONNECTIONS = '100' not 151 unless you changed it. Also, in truth you have MAX_CONNECTIONS + 1. The plus 1 allows a root user to logon even after you have maxed out the conenctions in order to figure out what is actually being used. When your connections are maxed out try logging on as root and running the following command from MySQL.
mysql> SHOW FULL PROCESSLIST
Post the output of this command above. Once you actually know what is consuming your resources you can go about fixing it.It could easily be your code that is leaving open connections.
You should take a look at the follwoing documentation: Show Processlist
+1 for question. Investigations showed us that node-mysql opens the connections and doesn't close them. Because of that at one moment be reach the limit of max connections. The question is why node-mysql doesn't close the connections?
We are migrating some websites onto a cloud infrastructure running Windows 2008 virtual machines. These websites all run on ColdFusion with MySQL databases. They currently are running in our CoLo with no problems. Additionally, they are running on our development network in our offices with no problems.
We are setting up our cloud to match as closely as possible the configuration we currently use which is, essentially, CF10 + IIS on one server and MySQL on a separate machine. We are 99% finished and most things are running great. However....
We have run into a couple, as in 2, places where we click a link/button and are greeted with:
Error Executing Database Query.
Communications link failure The last packet successfully received from the server was 0 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago.
Scanning the stack-trace I also find:
Caused by: java.net.SocketException: Connection reset
The communications link error is ALWAYS: 0ms.
What's most puzzling is the Queries that seem to be causing this are simple queries that are used ALL OVER the sites with no problems. Why they are failing at hese 2 particular places has us at wits end.
Our only clue is, looking at the CF Error description of what scripts are called, we can see the script where the query is failing is getting called twice? For example, one of the occurences is in our Application file:
>The error occurred in D:/Our_Web_Sites/oursite/Application.cfm: line 73
>Called from D:/Our_Web_Sites/oursite/Application.cfm: line 17
>Called from D:/Our_Web_Sites/oursite/Application.cfm: line 1
>Called from D:/Our_Web_Sites/oursite/Application.cfm: line 73
>Called from D:/Our_Web_Sites/oursite/Application.cfm: line 17
>Called from D:/Our_Web_Sites/oursite/Application.cfm: line 1
We can find nothing in our CF code that would be causing the script to be called twice so our guess is the first call is failing on the Query so CF tries again...only to fail and error.
Googling this issue I've found lots of posts about changing the MySQL timeouts. None of those worked and I didn't expect them to since what we're dealing with doesn't appear to be a timeout issue. These pages fail each and every time.
The closest we've come to a solution came from this blog posting:
http://www.talkingtree.com/blog/index.cfm/2011/1/12/Validation-Query-for-MySQL-communications-link-failure!
If we UNCHECK the "Maintain connections across client requests. " setting in CFAdmin then the error goes away. The blog suggests leaving that checked, which is our preference, and using Connection Validation of "SELECT 1;". Try that...same error.
We've also tried the JDBC AutoConnect=true option. No effect.
Downloaded latest JDBC Connector and used it instead of standard CF10-MySQL connector. No effect.
Again, 99% of the site works with the exception of these two links, both of which work just fine in all our other environments. Any other ideas?
I feel like I've had a similar problem every time I upgrade CF or MySQL. Usually a change in the JDBC driver or connection string helps, which I see you already tried.
Have you checked the MySQL error log for any hints? Ours is in /var/lib/mysql (whatever your 'datadir' variable is set to) and ends with a .err extension.
Also, maybe trying some of the other JDBC connection string options for your version? I see some extended logging you can enable.
http://dev.mysql.com/doc/refman/5.1/en/connector-j-reference-configuration-properties.html
Found the issue. We are running our network on Savvis' cloud infrastructure. The Windows server instances we were using from Savvis had Trend Micro Deep Security Agent installed. This is an intrusion protection system and it was the problem. Disabling the service cleared up all communication errors. I have no clue why it was rejecting some queries that it had just accepted previously. I am just glad to (finally) put this behind me!
I am confused with MySQL connections. I have site that receives heavy requests during working hours. I use PHP to connect to MySQL database using persistant connection.
Few weeks back, I increased mysql connections to 500 that crashed my server then I put it back to 150.
Now users complaints that sometimes they cannot get on the site. I believe that this is due to limited connections.
Can you please give me some information that whether I use persistant or non-persistant? What sections of mysql do I need to tune to get optimized connection processing?
I have attached a screenshot that shows 11K Failed Attempts.
http://i.stack.imgur.com/GkxHP.jpg
Thank you so much...
Update Dec 17, 2011
When I asked this question, I changed the connection type to "non-persistant" and everything starts working fine. Today I surprised to see that the stats from phpmyadmin. Below are the values given by Phpmyadmin:
max. concurrent connections :: 16
Failed Attempts :: 43k
Please suggest some possible solutions? Which parameter should be optimized to avoid/minimize Failed attempts?
High traffic sites should not use persistent connections. I changed DB connection from persistent to non-persistent in php and problem solved!
Thanks for your help.
EDIT:
After changing connection type to non-persistent, don't forget to increase number of connections. In my case, I increased them to 500 with type set to non-persistent and that solved the issue.