I am running a website on django 1.10, python3.4, mysqlclient, mysql5.5.
I have multiple management command running in the background for various tasks (like mail sending, updating tables) at different times. Recently, I have started seeing many Mysql gone away (Error 2006).
I tried to change the connection_timeout to be greater than max_connection age. That solved a small share of problem in which Error 2006 stopped to occur for a running function in most of the case but other cases are as it is and frequently seeing Error 2006.
In some cases, I tried using django.db.connection.close() before running a function or putting it in try - except loop. In those places, this error are not occuring anymore but then I cannot put this try-except loop in every function.
What is the root cause of this repeated error Mysql gone away? What are the different solution and what is the best solution so that I do not have to change much of my codes?
Other Variables:
We upgraded from django 1.7 to 1.10 and updated the mysqlclient package recently and the problem surfaced after that probably.
Just around the update, the traffic on our site also increased multiple times. Can that be a trigger?
As seen elsewhere, the problem can be either that your query is too long, or the server closed the connection due to inactivity. It depends a lot on the circumstances, but it seems that if you're not in the middle of a transaction (which would discard the second possibility), you can apply something similar to Hook available for automatic retry after deadlock in django and mysql setup
Related
Been working with the XAMP which I installed just under a year. I recently installed few frontend software for MySQL (to see which one am I comfortable with the most).
Now, for the past two days, whenever I go to localhost/phpmysql, I receive this warning
Fatal error: Maximum execution time of 30 seconds exceeded in C:\xampp\phpMyAdmin\lib..
I understand that the maximum execution time required to execute is being exceeded here. Found a few post on stackoverflow which guides you to rectification. All well and good till here.
I have a question and a concern.
Question
Why all of a sudden this error when, clearly remember, I did nothing to upset the default settings of the MySQL?
Concern
I am working on a project which uses a database (important, cannot loose it), the phpmyadmin when refreshed after the warning starts to work normally as there never was a problem. I'll need a couple of week to get done with my project. Can I continue with this timeout error without risking my database or should I try and rectify it right away?
DCoder answer is good, try to change the maximium execution time in MySQL server, also, you can try to find what query is making the noise using the slow-query-log (you can activate it and is very useful in the case of messy querys).
Check this page for more information
If you're using MySQL on windows environments, I strongly suggest to put it on a linux server, because it's lots slower over windows.
About your concern, There is no real danger about your data, and changing the slow-query-log option or maximum-execution-time option doesn't compromise the databases... but do some backup before change anything, just for extra security.
What is the effect of multiple tadoconnection?
Here is what I did :
I put a TADOConnection to almost every form in my application.
Those TADOConnection will connect to the database(MySQL) everytime I create an instance of a form.
In an average use of the application, about 15 forms will be used(15 tadoconnections connected to the database). So far my application is running smooth. But yesterday, a user complained of an error "MySQL has gone away".
I've encountered that error in the past and it was because the data is too large, or hardware problem. But today, the data is not big and the hardware is in excellent condition. By the way, the connection is local. Does the multiple tadoconnection produced the error?
The effect of multiple ADOConnections is that you, open multiple independent Session in the Database. I wouldnt recommend your solution, in consideration of Transactionmanagement and table locking
Server has gone away: http://dev.mysql.com/doc/refman/5.1/en/gone-away.html
One of the more interesting "features" in Coldfusion is how it handles external requests. The basic gist of it is that when a query is made to an external source through <cfquery> or or any other external request like that it passes the external request on to a specific driver and at that point CF itself is unable to suspend it. Even if a timeout is specified on the query or in the cfsetting it is flatly ignored for all external requests.
http://www.coldfusionmuse.com/index.cfm/2009/6/9/killing.threads
So with that in mind the issue we've run into is that somehow the communication between our CF server and our mySQL server sometimes goes awry and leaves behind hung threads. They have the following characteristics.
The hung thread shows up in CF and cannot be killed from FusionReactor.
There is no hung thread visible in mySQL, and no active running query (just the usual sleeps).
The database is responding to other calls and appears to be operating correctly.
Max connections have not been reached for the DB nor the user.
It seems to me the only likely candidate is that somehow CF is making a request, mySQL is responding to that request but with an answer which CF ignores and continues to keep the thread open waiting for a response from mySQL. That would explain why the database seems to show no signs of problems, but CF keeps a thread open waiting for the mysterious answer.
Usually these hung threads appear randomly on otherwise working scripts (such as posting a comment on a news article). Even while one thread is hung for that script, other requests for that script will go through, which would imply that the script isn't neccessarily at fault, but rather the condition faced when the script was executed.
We ran some test to determine that it was not a mysql generated max_connections error... we created a user, gave it 1 max connections, tied that connection with a sleep(1000) query and executed another query. Unfortunately, it correctly errored out without generating a hung thread.
So, I'm left at this point with absolutely no clue what is going wrong. Is there some other connection limit or timeout which could be causing the communication between the servers to go awry?
One of the things you should start to look at is the hardware between the two servers. It is possible that you have a router or bridge or NIC that is dropping occasional packets. This can result in the mySQL box thinking it has completed the task while the CF server sits there and waits for a complete response indefinitely, creating a hung thread.
3com has some details on testing for packet loss here: http://support.3com.com/infodeli/tools/netmgt/tncsunix/product/091500/c11ploss.htm#22128
We had a similar problem with a MS SQL server. There, the root cause was a known issue in which, for some reason, the server thinks it's shutting down, and the thread hangs (even though the server is, obviously, not shutting down).
We weren't able to eliminate the problem, but were able to reduce it by turning off pooled DB connections and fiddling with the connection refresh rate. (I think I got that label right -- no access to administrator at my new employment.) Both are in the connection properties in Administrator.
Just a note: The problem isn't entirely with CF. The problem, apparently, affects all Java apps. Which does not, in any way, reduce how annoyed I get by this.
Long story short, but I believe the caused was due to Coldfusion's CF8 image processing. It was just buggy and now in CF9 I have never seen that problem again.
I have a MySQL gone away with Django under WSGI. I found entries for this problem on stackoverflow, but nothing with Django specifically. Google does not help, except for workarounds (like polling the website every once in a while, or increasing the database timeout). Nothing definitive. Technically, Django and/or MySQLdb (I'm using the latest 1.2.3c1) should attempt a reconnect if the server hanged the connection, but this does not happen. How can I solve this issue without workarounds ?
show variables like 'wait_timeout';
this is the setting will throw back the "mysql gone away" error
set it to a very large value to prevent it "gone away"
or simple re-ping the mysql connection after certain period
Django developers gave one short answer for all questions like this in https://code.djangoproject.com/ticket/21597#comment:29
Resolution set to wontfix
Actually this is the intended behavior after #15119. See that ticket for the rationale.
If you hit this problem and don't want to understand what's going on, don't reopen this ticket, just do this:
RECOMMENDED SOLUTION: close the connection with from django.db import connection; connection.close() when you know that your program is going to be idle for a long time.
CRAPPY SOLUTION: increase wait_timeout so it's longer than the maximum idle time of your program.
In this context, idle time is the time between two successive database queries.
You could create middleware to ping() the MySQL connection (which
will reconnect if it timed out) before processing the view
You could also add middleware to catch the exception, reconnect, and retry the
view (I think I would prefer the above solution as simpler, but it should technically work and be performant assuming timeouts are rare. This also assumes a failed view has no side effects, which is a desirable property but can be difficult to do, especially if you write to a filesystem as well as a db in your view.)
I have a rails app that I have serving up XML on an infrequent basis.
This is being run with mongrel and mysql.
I've found that if I don't exercise the app for longer than a few hours it goes dead and starts throwing Errno::EPIPE errors. It seems that the mysql connection get timed out for inactivity or something like that.
It can be restarted with 'mongrel_rails restart -P /path/to/the/mongrel.pid' ... but that's not really a solution.
My collaborator expects the app to be there when he is working on his part (and I am most likely not around).
My question is:
What can I do to prevent this problem from occurring in the 1st place? (e.g. don't time me out!!).
Failing that, is there some code I can insert somewhere to automatically remake the Db connection?
Here's a solution:
https://boxpanel.blueboxgrp.com/public/the_vault/index.php/Mongrel_/_MySQL_Timeout
The timeouts on the above solution seem a little high to me. You don't want your DB timeouts to be too low, because of the amount of memory a connection can use. If a connection is orphaned, you want it to time out reasonably (like not in one week.)
In other places, I also got the following suggestions:
Try setting
config.active_record.verification_timeout to something lower than whatever
your mysql connection timeout setting is.
There's a gem to work around this problem: mysql_retry_lost_connection
http://rubyforge.org/projects/zventstools/
"Reconnect to the MySQL server when you hit a lost connection error".