I have a MySQL gone away with Django under WSGI. I found entries for this problem on stackoverflow, but nothing with Django specifically. Google does not help, except for workarounds (like polling the website every once in a while, or increasing the database timeout). Nothing definitive. Technically, Django and/or MySQLdb (I'm using the latest 1.2.3c1) should attempt a reconnect if the server hanged the connection, but this does not happen. How can I solve this issue without workarounds ?
show variables like 'wait_timeout';
this is the setting will throw back the "mysql gone away" error
set it to a very large value to prevent it "gone away"
or simple re-ping the mysql connection after certain period
Django developers gave one short answer for all questions like this in https://code.djangoproject.com/ticket/21597#comment:29
Resolution set to wontfix
Actually this is the intended behavior after #15119. See that ticket for the rationale.
If you hit this problem and don't want to understand what's going on, don't reopen this ticket, just do this:
RECOMMENDED SOLUTION: close the connection with from django.db import connection; connection.close() when you know that your program is going to be idle for a long time.
CRAPPY SOLUTION: increase wait_timeout so it's longer than the maximum idle time of your program.
In this context, idle time is the time between two successive database queries.
You could create middleware to ping() the MySQL connection (which
will reconnect if it timed out) before processing the view
You could also add middleware to catch the exception, reconnect, and retry the
view (I think I would prefer the above solution as simpler, but it should technically work and be performant assuming timeouts are rare. This also assumes a failed view has no side effects, which is a desirable property but can be difficult to do, especially if you write to a filesystem as well as a db in your view.)
Related
I don't set the variable CONN_MAX_AGE in my config, it is said it will be 0 as default, which means every request will reopen a new connection.But In my program it continuously shows the error (2006, server gone away) as every query starts.I don't know why.
You're not alone. This has been a known problem since 1.6, and I believe remains. (Happy to be corrected). Original documentation relating to the problem from the Django Project.
Broadly, I would reconfigure your problem so that you're opening a new database connection periodically. We have a number of long-running processes that connect to MySQL and found this to be a recurrent issue.
You have three options.
Break your function up slightly so that the chunks can each establish their own connection. This should be the implicit behaviour from the set-up you've described, but I've also run into problems with it.
from django import db
for thing in some_data:
db.connections.close_all()
your_stuff_here()
The explicit closing (and subsequent reopening) of the connection is also clearer than the implicit behaviour, which is also dependent on how your MySQL timeout is configured.
A very hacky workaround is to wrap your internal functions in a try/except:
from django.db.utils import OperationalError
try:
stuff = Stuff.objects.get(id=123456)
except OperationalError:
from django.db import connections
c = connections['default'].cursor()
stuff = Stuff.objects.get(id=123456)
This definitely works, as the cursor connection reopens it in the event of a failure. Although I implemented this as a fix on one of our long running processes, it's definitely a hack. I don't recommend it, and we subsequently refactored it out to the first method.
Lastly, you could up your MySQL timeout, which is noted on the django documentation page as a less-optimal solution.
https://dev.mysql.com/doc/refman/5.6/en/auto-reconnect.html
I'm using civetweb as a (websocket) server. I have some handlers for when I receive data, that will query mysql. These will spawn as a thread, every time there is a request.
Until now I was only using one mysql connection with the database, which I setup on the start off the program, in combination with the mongoose library. But with the threaded requests, it's causing me headaches, since mysql isn't thread-safe from the time you do mysql_select() until mysql_store_result(). I have tried putting a mutex around these mysql functions, but then perfomce drops a tenfold (from ~750 requests/second to ~75 requests/second).
What is the correct way to handle this? I've heard about a 'connection pool', but it's hard to find some simple examples with google (or wrap my head around a sane implementation).
It seems unlikely I'm the first person to encounter such a problem :).
I'm not sure if this will help you. I'd put it in a comment but I don't have enough reputation yet.
I had the same problem in VB.NET when I added multi threading to my application. To correct it there I made sure to call connection.open before all of my queries and added "pooling=true;" to the end of my mysql Connection String. Mysql will determine if it needs to open a connection or using existing one.
I'm doing my first foray with mysql and I have a doubt about how to handle the connection(s) my applications has.
What I am doing now is opening a connection and keeping it alive until I terminate my program. I do a mysql_ping() every now and then and the connection is started with MYSQL_OPT_RECONNECT.
The other option (I can think of), would be to start a new connection before doing anything that requires my connection to the database and closing it after I'm done with it.
What are the pros and cons of these two approaches?
what are the "side effects" of a long connection?
What is the most used method of handling this?
Cheers ;)
Some extra details
At this point I am keeping the connection alive and I ping it every now and again to now it's status and reconnect if needed.
In spite of this, when there is some consistent concurrency with queries happening in quick succession, I get a "Server has gone away" message and after a while the connection is re-established.
I'm left wondering if this is a side effect of a prolonged connection or if this is just a case of bad mysql server configuration.
Any ideas?
In general there is quite some amount of overhead incurred when opening a connection. Depending on how often you expect this to happen it might be ok, but if you are writing any kind of application that executes more than just a very few commands per program run, I would recommend a connection pool (for server type apps) or at least a single or very few connections from your standalone app to be kept open for some time and reused for multiple transactions.
That way you have better control over how many connections get opened at the application level, even before the database server gets involved. This is a service an application server offers you, but it can also be rolled up rather easily if you want to keep it smaller.
Apart from performance reasons a pool is also a good idea to be prepared for peaks in demand. When a lot of requests come in and each of them tries to open a separate connection to the database - or as you suggested even more (per transaction) - you are quickly going to run out of resources. Keep in mind that every connection consumes memory inside MySQL!
Also you want to make sure to use a non-root user to connect, because if you don't (I think it is tied to the MySQL SUPER privilege), you might find yourself locked out. MySQL reserves at least one connection for an administrator for problem fixing, but if your app connects with that privilege, all connections would already be used up when you try to put out the fire manually.
Unless you are worried about having too many connections open (i.e. over 1,000), you she leave the connection open. There is overhead in connecting/reconnecting that will only slow things down. If you know you are going to need the connection to stay open for a while, run this query instead of pinging periodically:
SET SESSION wait_timeout=#
Where # is the number of seconds to leave an idle connection open.
What kind of application are you writing? If it's a webscript: keep it open. If it's an executable, pool your connections (if necessary, most of the times a singleton will do).
One of the more interesting "features" in Coldfusion is how it handles external requests. The basic gist of it is that when a query is made to an external source through <cfquery> or or any other external request like that it passes the external request on to a specific driver and at that point CF itself is unable to suspend it. Even if a timeout is specified on the query or in the cfsetting it is flatly ignored for all external requests.
http://www.coldfusionmuse.com/index.cfm/2009/6/9/killing.threads
So with that in mind the issue we've run into is that somehow the communication between our CF server and our mySQL server sometimes goes awry and leaves behind hung threads. They have the following characteristics.
The hung thread shows up in CF and cannot be killed from FusionReactor.
There is no hung thread visible in mySQL, and no active running query (just the usual sleeps).
The database is responding to other calls and appears to be operating correctly.
Max connections have not been reached for the DB nor the user.
It seems to me the only likely candidate is that somehow CF is making a request, mySQL is responding to that request but with an answer which CF ignores and continues to keep the thread open waiting for a response from mySQL. That would explain why the database seems to show no signs of problems, but CF keeps a thread open waiting for the mysterious answer.
Usually these hung threads appear randomly on otherwise working scripts (such as posting a comment on a news article). Even while one thread is hung for that script, other requests for that script will go through, which would imply that the script isn't neccessarily at fault, but rather the condition faced when the script was executed.
We ran some test to determine that it was not a mysql generated max_connections error... we created a user, gave it 1 max connections, tied that connection with a sleep(1000) query and executed another query. Unfortunately, it correctly errored out without generating a hung thread.
So, I'm left at this point with absolutely no clue what is going wrong. Is there some other connection limit or timeout which could be causing the communication between the servers to go awry?
One of the things you should start to look at is the hardware between the two servers. It is possible that you have a router or bridge or NIC that is dropping occasional packets. This can result in the mySQL box thinking it has completed the task while the CF server sits there and waits for a complete response indefinitely, creating a hung thread.
3com has some details on testing for packet loss here: http://support.3com.com/infodeli/tools/netmgt/tncsunix/product/091500/c11ploss.htm#22128
We had a similar problem with a MS SQL server. There, the root cause was a known issue in which, for some reason, the server thinks it's shutting down, and the thread hangs (even though the server is, obviously, not shutting down).
We weren't able to eliminate the problem, but were able to reduce it by turning off pooled DB connections and fiddling with the connection refresh rate. (I think I got that label right -- no access to administrator at my new employment.) Both are in the connection properties in Administrator.
Just a note: The problem isn't entirely with CF. The problem, apparently, affects all Java apps. Which does not, in any way, reduce how annoyed I get by this.
Long story short, but I believe the caused was due to Coldfusion's CF8 image processing. It was just buggy and now in CF9 I have never seen that problem again.
I have a rails app that I have serving up XML on an infrequent basis.
This is being run with mongrel and mysql.
I've found that if I don't exercise the app for longer than a few hours it goes dead and starts throwing Errno::EPIPE errors. It seems that the mysql connection get timed out for inactivity or something like that.
It can be restarted with 'mongrel_rails restart -P /path/to/the/mongrel.pid' ... but that's not really a solution.
My collaborator expects the app to be there when he is working on his part (and I am most likely not around).
My question is:
What can I do to prevent this problem from occurring in the 1st place? (e.g. don't time me out!!).
Failing that, is there some code I can insert somewhere to automatically remake the Db connection?
Here's a solution:
https://boxpanel.blueboxgrp.com/public/the_vault/index.php/Mongrel_/_MySQL_Timeout
The timeouts on the above solution seem a little high to me. You don't want your DB timeouts to be too low, because of the amount of memory a connection can use. If a connection is orphaned, you want it to time out reasonably (like not in one week.)
In other places, I also got the following suggestions:
Try setting
config.active_record.verification_timeout to something lower than whatever
your mysql connection timeout setting is.
There's a gem to work around this problem: mysql_retry_lost_connection
http://rubyforge.org/projects/zventstools/
"Reconnect to the MySQL server when you hit a lost connection error".