I have SQLAlchemy CORE 1.0.9 with Pyramid framework 1.7. And I am using the following configuration for connecting to a postgres 9.4 database:
# file __ini__.py
from .factories import root_factory
from pyramid.config import Configurator
from sqlalchemy import engine_from_config
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application."""
config = Configurator(settings=settings, root_factory=root_factory)
engine = engine_from_config(settings, prefix='sqlalchemy.')
# Retrieves database connection
def get_db(request):
connection = engine.connect()
def disconnect(request):
connection.close()
request.add_finished_callback(disconnect)
return connection
config.add_request_method(get_db, 'db', reify=True)
config.scan()
return config.make_wsgi_app()
After a few hours using the app I start getting the following error:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: remaining connection slots are reserved for non-replication superuser connections
Apparently I have reached the maximum number of connections. It seems like connections.close() doesn't really close the connection, just returns the connection to the pool. I know I could use the NullPool to disable pooling but probably that will have a huge impact in the performance.
Does somebody know the right way to configure SQLAlchemy Core to get a good performance and close the connections properly?
Please abstain from sending links to pyramid tutorials. I am NOT interested in SQLAlchemy ORM setups. Only SQLAlchemy Core please.
Actually everything was fine in the previous setup. The problem was caused by Celery workers not closing connections.
Related
I'm newbie running a Flask app connected to a MySQL remote server with Flask-SQLAlchemy.
The app has very little traffic and it's usual to stay idle for more than 8 hours, I then get disconnected from the MySQL server.
this is my app code:
from flask import Flask, render_template, request, redirect, jsonify, make_response
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import or_
app = Flask(__name__)
app.config['DEBUG'] = True
app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {'pool_pre_ping': True, 'pool_recycle': 300, 'echo':'debug',}
app.config['SQLALCHEMY_DATABASE_URI']='mysql+pymysql://user:pass#myipcode/db'
db = SQLAlchemy(app)
Everything works ok until no querys are performed for 8 hours, then I lose db connection and logs show this error code:
"MySQL server has gone away (%r)" % (e,)) sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))
I did some research and was adviced to set 'SQLALCHEMY_ENGINE_OPTIONS' as written in the code sample but it behaves the same with or without such engine options, connection is not recycled every 300 seconds as It should, and pool_pre_ping doesn't seem to make any difference. 'echo':'debug' option works as intended since I get every transaction logged.
What should I do to prevent the connection from being disconnected even after a long period of inactivty?
EDIT:
To add some additional info:
The database is hosted in Cloud SQL from GCP .
I'm lost...
Any help would be greatly appreciated.
Finally I figured it out myself,
It had to do with the fact that my app was running on a mountpoint like http://ServerIP/app instead of http://ServerIP/ because it was initially intended as a staging server.
I was using uWSGI and, in order to make it work in the aforementioned path, I had to specify a mount parameter in the [UWSGI] block from app.ini file.
When the uWSGI server started it showed like it was mounting two apps, one in '' and the other in '/app', and I guess that created a conflict that made the app unable to manage connections on MySQL server.
Mounting the app in http://ServerIP/ worked like a charm.
I have a Django App with a pretty standard server stack
DB Backend : MySQL
WSGI Server : Gunicorn
Async worker class : Gevent
I want Django to pool MySQL connections rather than creating connections on every request.
Starting 1.6, Django has introduced persistent connections but there are issues with async workers.
Hence, either a different MySQL backend is required or app level connection pooling. I've read several of them. Some of them are very old articles. Following are some:
Django MySQL backends
django-mysqlpool
App level Connection pool
with SQL Alchemy
another with SQL Alchemy
Some Patches are also available
Django Patch
Some other approaches
MySQL DB Connector
I'm really confused as to which approach among these is the best way to pool connections? Any Help is highly appreciated.
This project still works on Django 1.9, and worked well for us.
https://github.com/djangonauts/djorm-ext-pool
your demand
want pool MySQL connections rather than creating connections on
every request.
my suggest
in db level
Indicating that your application is IO-intensive, so the proposal
is to use mysql conn pool. may be u can use thirdpart mysql pool
in app level
in app level no use connection pooling. But mostly use the cache
,may be redis cache etc,this can minus the connection number.
in webserver level
in your server socalled WSGI Server . It is ligntweight so not
pooling implement,u can refact to use queue to enhance the connection
reused. or base Gevent to refact event_queue.
Hope this may can give you some help.
I'm running and pyramid app inside gunicorn container with gevent async workers,
one of endpoints is a long-pool endpoint pooling AMQP via kombu.
If the long pool withing 30s timeout returns some data from AMQP I need to save it to postgres before returning, now the question is:
Is it OK to start concurrent.futures.ThreadPoolExecutor in context on app running in gevent loop, and deal with SQLAlchemy sessions and data persistence inside a future submitted to executor?
Or am I completely wrong in my way of thinking?
PS DB driver is psycopg2
After using ThreadPoolExecutor with gevent in my high-concurrent app I can confirm that this works fine.
But unnecessary if the DB Driver is already gevent "friendly" like: psycogreen
I've inherited an application making use of python & sqlalchemy to interact with a mysql database. When I issue:
mysql_engine = sqlalchemy.create_engine('mysql://uname:pwd#192.168.xx.xx:3306/testdb', connect_args={'use_unicode':True,'charset':'utf8', 'init_command':'SET NAMES UTF8'}, poolclass=NullPool)
, at startup, an exception is thrown:
cmd = unicode("USE testdb")
with mysql_engine.begin() as conn:
conn.execute(cmd)
sqlalchemy.exc.OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.168.xx.xx' (101)") None None
However, using IDLE I can do:
>>> import MySQLdb
>>> Con = MySQLdb.Connect(host="192.168.xx.xx", port=3306, user="uname", passwd="pwd", db="testdb")
>>> Cursor = Con.cursor()
>>> sql = "USE testdb"
>>> Cursor.execute(sql)
The application at this point defaults to using an onboard sqlite database. After this I can quite happily switch to the MySQL database using the create_engine statement above. However, on reboot the MySQL database connection will fail again, defaulting to the onboard sqlite db, etc, etc.
Has anyone got any suggestions as to how this could be happening?
Just thought I would update this - the problem still occurs exactly as described above. I've updated the app so that the user can manually connect to the MySQL db by selecting a menu option. This calls the identical code which exceptions when the app is starting, but works just fine once the app is up and running.
The MySQL instance is completely separate from the app and running throughout, so it should be available to receive connections at all times.
I guess the fundamental question i'm grappling with is how can the same connect code work when the app is up and running, but throw an exception when it is starting?
Is there any artifact of SQLAlchemy that can cause it to fail to create usable connections that isn't dependant on the connection parameters or the remote database?
Ahhh, it all seems so obvious now...
The reason for the exception on startup was because the network interface hadn't finished configuring when the application would make its first request to the remote database. (Which is why the same thing would be successful when attempted at a later time).
As communication with the remote database is a prerequisite for the application, I now do something like this:
if grep -Fxq "mysql" /path/to/my/db/config.config
then
while ! ip a | grep inet.*wlan0 ; do sleep 1; echo "waiting for network..."; done;
fi
... in the startup script for my application - ensuring that the network interface has finished configuring before the application can run.
Of course, the application will never run if the interface doesn't configure, so it still needs some finessing to allow it to timeout and default to using a local database...
I am currently using Flask-uWSGI-Websockets to provide websocket functionality for my application. I use Flask-SQLAlchemy to connect to my MySQL database.
Flask-uWSGI-Websockets uses gevent to manage websocket connections.
The problem I am currently having is that when a websocket connection is ended, the database connection set up by Flask-SQLAlchemy will keep on living.
I have tried calling db.session.close() and db.engine.dispose() after every websocket connection, but this had no effect.
Calling gevent.monkey.patch_all() at the beginning of my app does not make a difference.
A simple representation of what I am doing is this:
from gevent.monkey import patch_all
patch_all()
from flask import Flask
from flask_uwsgi_websocket import GeventWebSocket
from flask_sqlalchemy import SQLAlchemy
app = Flask()
ws = GeventWebSocket()
db = SQLAlchemy()
db.init_app(app)
ws.init_app(app)
#ws.route('/ws')
def websocket(client):
""" handle messages """
while client.connected is True:
msg = client.recv()
# do some db stuff with the message
# The following part is executed when the connection is broken,
# i tried this for removing the connection, but the actual
# connection will stay open (i can see this on the mysql server).
db.session.close()
db.engine.dispose()
I have same situation. and solution for me located in mysql configuration file my.cnf:
[mysqld]
interactive_timeout=180
wait_timeout=180
you must restart mysql service after save my.cnf.
if you don't want to restart mysql service you can use sql queries:
SET GLOBAL interactive_timeout = 180;
SET GLOBAL wait_timeout = 180;
See also wait_timeout and interactive_timeout on mysql.com