I successfully uploaded my webapp onto webfaction, but I noticed, when using peewee to connect to a MySQL database on my webfaction account, I got this error:
ProgrammingError: (1146, "Table 'TABLEGOESHERE' doesn't exist")
Exact error is in the error log file below
Some background information:
I created a MySQL database on webfaction
I did not create any tables within the control panel provided by the service. It's completely empty.
I can run my flask app successfully through the terminal, but I am about to make it live on a web server, and so I am very new to this process.
I assumed that when you're using peewee you can create tables from within your program like so:
models.py
# -- Peewe Modules
from peewee import *
DATABASE = MySQLDatabase("DBNAMEGOESHERE", host="HOSTGOESHERE", port=PORTGOESHERE, user="USERGOESHERE", passwd="PASSGOESHERE")
# -- DATABASE OBJECTS GO HERE:
#-- INIT
def initialize():
DATABASE.connect()
DATABASE.create_tables([Post, etc...],safe=True)
DATABASE.close()
The initialize function is called in the __init__.py file at the bottom of the file like so:
if __name__ == "__main__":
models.initialize()
try:
models.User.create_user(
username = 'user',
email = 'email',
password = 'pass',
is_admin = True,
confirmed = True,
confirmed_on = datetime.datetime.now(),
)
except ValueError:
pass
app.run()
My index view, which routes to ('/'), in my __init__.py file makes a call to the count method like so:
count = models.Post.select().count()
And I believe this line caused my site to display a 500 internal server error which resulted in this error log (Timestamps have been removed for simplicity):
return self.wsgi_app(environ, start_response)
File "/home/username/webapps/myapp/myapp/__init__.py", line 49, in __call__
return self.app(environ, start_response)
File "/home/username/lib/python2.7/Flask-0.10.1-py2.7.egg/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/home/username/lib/python2.7/Flask-0.10.1-py2.7.egg/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/username/lib/python2.7/Flask-0.10.1-py2.7.egg/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/username/lib/python2.7/Flask-0.10.1-py2.7.egg/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/username/lib/python2.7/Flask-0.10.1-py2.7.egg/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/username/lib/python2.7/Flask-0.10.1-py2.7.egg/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/username/lib/python2.7/Flask-0.10.1-py2.7.egg/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/username/webapps/myapp/myapp/__init__.py", line 587, in index
count = models.Post.select().count()
File "/home/username/lib/python2.7/peewee-2.7.3-py2.7.egg/peewee.py", line 2792, in count
return self.aggregate(convert=False) or 0
File "/home/username/lib/python2.7/peewee-2.7.3-py2.7.egg/peewee.py", line 2785, in aggregate
return self._aggregate(aggregation).scalar(convert=convert)
File "/home/username/lib/python2.7/peewee-2.7.3-py2.7.egg/peewee.py", line 2564, in scalar
row = self._execute().fetchone()
File "/home/username/lib/python2.7/peewee-2.7.3-py2.7.egg/peewee.py", line 2555, in _execute
return self.database.execute_sql(sql, params, self.require_commit)
File "/home/username/lib/python2.7/peewee-2.7.3-py2.7.egg/peewee.py", line 3366, in execute_sql
self.commit()
File "/home/username/lib/python2.7/peewee-2.7.3-py2.7.egg/peewee.py", line 3212, in __exit__
reraise(new_type, new_type(*exc_args), traceback)
File "/home/username/lib/python2.7/peewee-2.7.3-py2.7.egg/peewee.py", line 3359, in execute_sql
cursor.execute(sql, params or ())
File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
ProgrammingError: (1146, "Table 'DATABASENAMEHERE.post' doesn't exist")
Can anyone help me identify and fix this issue? I have no idea how to get my flask app to cooperate with my MySQL database on webfaction.
Are you sure your application is being run by executing it directly from the command-line? i.e. the __name__ == '__main__' block is actually running? Is it possible you're using a dedicated WSGI server instead?
Moved my server initialization code into a file that runs the Flask app on Webfaction. It is not recommended to place additional code under the conditional as its meant for the command line.
Related
I have a Flask website that uses the peewee ORM. The connection is managed by FlaskDB.
When I only use 1 gunicorn worker, it works well. But as soon as I use 2 or more gunicorn workers, I start getting this error:
Traceback (most recent call last):
File "/home/user/.local/lib/python3.10/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/home/user/.local/lib/python3.10/site-packages/flask/app.py", line 1519, in full_dispatch_request
return self.finalize_request(rv)
File "/home/user/.local/lib/python3.10/site-packages/flask/app.py", line 1540, in finalize_request
response = self.process_response(response)
File "/home/user/.local/lib/python3.10/site-packages/flask/app.py", line 1888, in process_response
self.session_interface.save_session(self, ctx.session, response)
File "/home/user/project/session.py", line 113, in save_session
saved_session.save()
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 6497, in save
rows = self.update(**field_dict).where(self._pk_expr()).execute()
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 1886, in inner
return method(self, database, *args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 1957, in execute
return self._execute(database)
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 2442, in _execute
cursor = database.execute(self)
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 3112, in execute
return self.execute_sql(sql, params, commit=commit)
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 3096, in execute_sql
with __exception_wrapper__:
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 2873, in __exit__
reraise(new_type, new_type(exc_value, *exc_args), traceback)
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 183, in reraise
raise value.with_traceback(tb)
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 3099, in execute_sql
cursor.execute(sql, params or ())
peewee.DatabaseError: error with status PGRES_TUPLES_OK and no message from the libpq
The peewee documentation states that the connection should be thread-safe, but it seems there are thread-safety issues here.
I use playhouse.pool.PooledPostgresqlDatabase if that matters.
What is the solution to this problem?
I believe this is possibly due to the connection being opened before the workers are forked. I'm not too familiar w/gunicorns worker model, but googling your error reveals similar problems w/multiprocessing. Specifically,
When a program uses multiprocessing or fork(), and an Engine object is copied to the child process, Engine.dispose() should be called so that the engine creates brand new database connections local to that fork. Database connections generally do not travel across process boundaries.
That is for SQLAlchemy, but the same should apply to Peewee. Just ensure that all your connections are closed (pool.close_all()) when your workers first start running. Or similarly, if you're opening any database connections at module scope, ensure you call close_all() after using them. This way when your workers start up they will each have an empty pool of connections.
db = FlaskDB(app)
# ...
db.database.close_all()
We added a second airflow database to support our staging airflow instance, and now both staging and production seem to have intermittent connection issues. We recently upgraded to 2.2.4, but looking through past RDS logs, I see aborted connections (Got an error reading communication packets) prior to the upgrade as well (previous version 1.10.11). Both instances have a webserver UI that pull in past DAG runs fine, and the scheduler is running DAGs appropriately and storing dag_run data. When the service is started however, we receive an error "MySQL server has gone away" (see below for stack trace). airflow db check and airflow db shell both return successful connections. Some relevant airflow config values are:
sql_alchemy_pool_enabled = True
sql_alchemy_pool_size = 5
sql_alchemy_max_overflow = 10
sql_alchemy_pool_recycle = 1800
sql_alchemy_pool_pre_ping = True
sql_alchemy_schema =
sql_alchemy_reconnect_timeout = 300
I've also tried upping the pool size to 20, and also confirmed that our db size should be able to handle at least 90 concurrent connections.
[2022-04-28 16:32:03,212] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create Admin
Process ForkProcess-19:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 254, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (2006, 'MySQL server has gone away')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/dag_processing/manager.py", line 287, in _run_processor_manager
processor_manager.start()
File "/usr/local/lib/python3.6/site-packages/airflow/dag_processing/manager.py", line 520, in start
return self._run_parsing_loop()
File "/usr/local/lib/python3.6/site-packages/airflow/dag_processing/manager.py", line 585, in _run_parsing_loop
self._find_zombies()
File "/usr/local/lib/python3.6/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/dag_processing/manager.py", line 1079, in _find_zombies
LJ.latest_heartbeat < limit_dttm,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3373, in all
return list(self)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 254, in query
_mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (2006, 'MySQL server has gone away')
[SQL: SELECT task_instance.try_number AS task_instance_try_number, task_instance.task_id AS task_instance_task_id, task_instance.dag_id AS task_instance_dag_id, task_instance.run_id AS task_instance_run_id, task_instance.start_date AS task_instance_start_date, task_instance.end_date AS task_instance_end_date, task_instance.duration AS task_instance_duration, task_instance.state AS task_instance_state, task_instance.max_tries AS task_instance_max_tries, task_instance.hostname AS task_instance_hostname, task_instance.unixname AS task_instance_unixname, task_instance.job_id AS task_instance_job_id, task_instance.pool AS task_instance_pool, task_instance.pool_slots AS task_instance_pool_slots, task_instance.queue AS task_instance_queue, task_instance.priority_weight AS task_instance_priority_weight, task_instance.operator AS task_instance_operator, task_instance.queued_dttm AS task_instance_queued_dttm, task_instance.queued_by_job_id AS task_instance_queued_by_job_id, task_instance.pid AS task_instance_pid, task_instance.executor_config AS task_instance_executor_config, task_instance.external_executor_id AS task_instance_external_executor_id, task_instance.trigger_id AS task_instance_trigger_id, task_instance.trigger_timeout AS task_instance_trigger_timeout, task_instance.next_method AS task_instance_next_method, task_instance.next_kwargs AS task_instance_next_kwargs, dag.fileloc AS dag_fileloc, dag_run_1.state AS dag_run_1_state, dag_run_1.id AS dag_run_1_id, dag_run_1.dag_id AS dag_run_1_dag_id, dag_run_1.queued_at AS dag_run_1_queued_at, dag_run_1.execution_date AS dag_run_1_execution_date, dag_run_1.start_date AS dag_run_1_start_date, dag_run_1.end_date AS dag_run_1_end_date, dag_run_1.run_id AS dag_run_1_run_id, dag_run_1.creating_job_id AS dag_run_1_creating_job_id, dag_run_1.external_trigger AS dag_run_1_external_trigger, dag_run_1.run_type AS dag_run_1_run_type, dag_run_1.conf AS dag_run_1_conf, dag_run_1.data_interval_start AS dag_run_1_data_interval_start, dag_run_1.data_interval_end AS dag_run_1_data_interval_end, dag_run_1.last_scheduling_decision AS dag_run_1_last_scheduling_decision, dag_run_1.dag_hash AS dag_run_1_dag_hash
FROM task_instance INNER JOIN job ON task_instance.job_id = job.id AND job.job_type IN (%s) INNER JOIN dag ON task_instance.dag_id = dag.dag_id INNER JOIN dag_run AS dag_run_1 ON dag_run_1.dag_id = task_instance.dag_id AND dag_run_1.run_id = task_instance.run_id
WHERE task_instance.state = %s AND (job.state != %s OR job.latest_heartbeat < %s)]
[parameters: ('LocalTaskJob', <TaskInstanceState.RUNNING: 'running'>, <TaskInstanceState.RUNNING: 'running'>, datetime.datetime(2022, 4, 28, 16, 27, 2, 870459))]
(Background on this error at: http://sqlalche.me/e/13/e3q8)```
I am running a Flask application on local and production server. I have no issues with the local, I am facing 'MySQL connection not available' for every second database request on production server. After reloading or after performing rollback operation, it is getting executed, but the issue repeats. I tried major solutions available on the internet by changing, pool_recycle, 'wait_timeout' and other timeouts to same value of 1600, but didn't work. I finally destroyed the application and reinstalled everything on the server but the issue is still on. Please help in this, thank you.
My production MySQL config:
class ProductionConfig(Config):
SECRET_KEY = 'secret_key'
DEBUG=False
SQLALCHEMY_DATABASE_URI = "mysql+pymysql://{username}:
{password}#{hostname}/{databasename}".format(
username="myusername",
password="password",
hostname="myhostname",
databasename="mydatabase",
)
SQLALCHEMY_POOL_RECYCLE = 299
SQLALCHEMY_TRACK_MODIFICATIONS = False
Error Message:
Exception on /add_questions/1/2 [POST]
Traceback (most recent call last):
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 713, in _write_bytes
self._sock.sendall(data)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
cursor, statement, parameters, context
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
cursor.execute(statement, parameters)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 516, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 771, in _execute_command
self._write_bytes(packet)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 718, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask_login/utils.py", line 270, in decorated_view
elif not current_user.is_authenticated:
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/werkzeug/local.py", line 347, in __getattr__
return getattr(self._get_current_object(), name)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/werkzeug/local.py", line 306, in _get_current_object
return self.__local()
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask_login/utils.py", line 26, in <lambda>
current_user = LocalProxy(lambda: _get_user())
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask_login/utils.py", line 346, in _get_user
current_app.login_manager._load_user()
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask_login/login_manager.py", line 318, in _load_user
user = self._user_callback(user_id)
File "/home/ulznrcvr/jaihindpro/application/auth/views.py", line 26, in load_user
return User.query.get(int(user_id))
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 1004, in get
return self._get_impl(ident, loading.load_on_pk_identity)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 1121, in _get_impl
return db_load_fn(self, primary_key_identity)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 287, in load_on_pk_identity
return q.one()
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3360, in one
ret = self.one_or_none()
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3329, in one_or_none
ret = list(self)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3405, in __iter__
return self._execute_and_instances(context)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3430, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 984, in execute
return meth(self, multiparams, params)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1103, in _execute_clauseelement
distilled_params,
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1288, in _execute_context
e, statement, parameters, cursor, context
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1482, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
cursor, statement, parameters, context
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
cursor.execute(statement, parameters)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 516, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 771, in _execute_command
self._write_bytes(packet)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 718, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
[SQL: SELECT user.id AS user_id, user.student_id AS user_student_id, user.role_id AS user_role_id, user.course_id AS user_course_id, user.sub_course_id AS user_sub_course_id, user.batch_id AS user_batch_id, user.full_name AS user_full_name, user.email AS user_email, user.phone_number AS user_phone_number, user.password AS user_password, user.active AS user_active, user.confirmed AS user_confirmed, user.confirmed_date AS user_confirmed_date, user.subscribed AS user_subscribed, user.start_date AS user_start_date, user.end_date AS user_end_date, user.re_url AS user_re_url, user.online AS user_online
FROM user
WHERE user.id = %(param_1)s]
[parameters: {'param_1': 2}]
(Background on this error at: http://sqlalche.me/e/e3q8)
I figured it out myself, I flagged "pool_pre_ping" attribute to "True" in engine settings.
SQLALCHEMY_ENGINE_OPTIONS = {
"pool_pre_ping": True,
"pool_recycle": 300,
}
Ref: https://medium.com/#heyjcmc/controlling-the-flask-sqlalchemy-engine-a0f8fae15b47
I have an airflow instance on EC2 that is running the webserver/scheduler. I want to hook up a MySQL RDS instance as the backend metadata database as opposed to the native SQLite. I replaced the one line in Airflow.cfg that connects to the database via sql_alchemy to connect to RDS with a pymysql driver:
#sql_alchemy_conn = sqlite:////home/cloud-user/airflow/airflow.db
sql_alchemy_conn = mysql+pymysql://admin:<PASSWORD>#airflow-db.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com:3306/airflow
The connection seems to work fine and I am able to get into the RDS instance and query tables via a MySQL client set up on my EC2 instance.
When I toggle a DAG on or off, I get this nasty python stack trace in my shell:
[2019-09-30 14:00:51,774] {app.py:1891} ERROR - Exception on /admin/airflow/paused [POST]
Traceback (most recent call last):
File "/home/cloud-user/.local/lib/python2.7/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/home/cloud-user/.local/lib/python2.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/cloud-user/.local/lib/python2.7/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/cloud-user/.local/lib/python2.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/home/cloud-user/.local/lib/python2.7/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/cloud-user/.local/lib/python2.7/site-packages/flask_admin/base.py", line 69, in inner
return self._run_view(f, *args, **kwargs)
File "/home/cloud-user/.local/lib/python2.7/site-packages/flask_admin/base.py", line 368, in _run_view
return fn(self, *args, **kwargs)
File "/home/cloud-user/.local/lib/python2.7/site-packages/flask_login/utils.py", line 258, in decorated_view
return func(*args, **kwargs)
File "/home/cloud-user/.local/lib/python2.7/site-packages/airflow/www/utils.py", line 279, in wrapper
session.commit()
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1027, in commit
self.transaction.commit()
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 494, in commit
self._prepare_impl()
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 473, in _prepare_impl
self.session.flush()
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2459, in flush
self._flush(objects)
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2597, in _flush
transaction.rollback(_capture_exception=True)
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2557, in _flush
flush_context.execute()
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
uow,
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
insert,
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1138, in _emit_insert_statements
statement, params
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 988, in execute
return meth(self, multiparams, params)
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
distilled_params,
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1253, in _execute_context
e, statement, parameters, cursor, context
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1473, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
cursor, statement, parameters, context
File "/home/cloud-user/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
cursor.execute(statement, parameters)
File "/home/cloud-user/.local/lib/python2.7/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/home/cloud-user/.local/lib/python2.7/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/home/cloud-user/.local/lib/python2.7/site-packages/pymysql/connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/home/cloud-user/.local/lib/python2.7/site-packages/pymysql/connections.py", line 732, in _read_query_result
result.read()
File "/home/cloud-user/.local/lib/python2.7/site-packages/pymysql/connections.py", line 1075, in read
first_packet = self.connection._read_packet()
File "/home/cloud-user/.local/lib/python2.7/site-packages/pymysql/connections.py", line 684, in _read_packet
packet.check_error()
File "/home/cloud-user/.local/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error
err.raise_mysql_exception(self._data)
File "/home/cloud-user/.local/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
raise errorclass(errno, errval)
ProgrammingError: (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[(\\'is_paused\\', u\\'true\\'), (\\'dag_id\\', u\\'test\\')]'')' at line 1")
[SQL: INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES (%(dttm)s, %(dag_id)s, %(task_id)s, %(event)s, %(execution_date)s, %(owner)s, %(extra)s)]
[parameters: {'task_id': None, 'extra': "[('is_paused', u'true'), ('dag_id', u'test')]", 'execution_date': None, 'event': 'paused', 'owner': 'anonymous', 'dttm': datetime.datetime(2019, 9, 30, 18, 0, 51, 768073, tzinfo=<Timezone [UTC]>), 'dag_id': u'test'}]
(Background on this error at: http://sqlalche.me/e/f405)
From what I saw on the Airflow documentation, I'm making the change correctly, and the 'airflow initdb / resetdb' commands execute without error.
I already spent a fair amount of time googling this error, but there are no clear answers to this problem. I'm really not sure if I'm missing a prerequisite or if I should be using a different connector?
EDIT: I'm using python 2.7, as seen in the stack trace. Airflow claims compatibility in the short-term, but I see another SO user's problems went away after upgrading to python 3.6: Link to other solution. I'll try this and update if it seems to work.
It looks like the solution is indeed to upgrade to python 3.6, leveraging a virtual environment due to the required duality of python 2.x and 3.y with Linux systems and Airflow. Specifically, I followed this guide and my DAGs seem to be executing successfully.
I am using SQLAlchemy 0.8 with Google App Engine and Cloud SQL. I would like to do a full integration test locally before deploying, but I cannot get SQLAlchemy to connect to my development MySQL server using the mysql+gaerdbms dialect
create_engine(
'mysql+gaerdbms:///test?instance=homingbox:instance1'
)
I start my app engine from the command line as follows
python dev_appserver.py --mysql_user=username --mysql_password=root ../Source/HomingBox/gae
Whenever I try to hit the database I get this lovely stack trace
ERROR 2013-04-22 21:35:39,987 webapp2.py:1528] (ImportError) None None
Traceback (most recent call last):
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/home/leon/Development/Source/HomingBox/gae/rest.py", line 19, in get_list
self.response.write(json.dumps(application.get_list()))
File "/home/leon/Development/Source/HomingBox/gae/dal.py", line 98, in get_list
return query.all()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/query.py", line 2140, in all
return list(self)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/query.py", line 2252, in __iter__
return self._execute_and_instances(context)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/query.py", line 2265, in _execute_and_instances
close_with_result=True)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/query.py", line 2256, in _connection_from_session
**kw)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/session.py", line 797, in connection
close_with_result=close_with_result)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/session.py", line 801, in _connection_for_bind
return self.transaction._connection_for_bind(engine)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/session.py", line 297, in _connection_for_bind
conn = bind.contextual_connect()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/engine/base.py", line 1669, in contextual_connect
self.pool.connect(),
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 272, in connect
return _ConnectionFairy(self).checkout()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 425, in __init__
rec = self._connection_record = pool._do_get()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 855, in _do_get
return self._create_connection()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 225, in _create_connection
return _ConnectionRecord(self)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 318, in __init__
self.connection = self.__connect()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 368, in __connect
connection = self.__pool._creator()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/engine/strategies.py", line 80, in connect
return dialect.connect(*cargs, **cparams)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/engine/default.py", line 279, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 183, in __init__
super(GoogleApiConnection, self).__init__(*args, **kwargs)
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms.py", line 810, in __init__
self.OpenConnection()
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms.py", line 832, in OpenConnection
self.SetupClient()
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 193, in SetupClient
self._client = RdbmsGoogleApiClient(**kwargs)
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 106, in __init__
rdbms.OAUTH_CREDENTIALS_PATH)
File "/usr/lib/python2.7/posixpath.py", line 268, in expanduser
import pwd
File "/home/leon/Development/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 827, in load_module
raise ImportError
Any help is much appreciated
Here's my hypothesis, based on the stack trace -- when the application can't find your HOME directory, it attempts to import pwd and get the home directory information that way. The application is trying to call this bit of code in posixpath.py:
if 'HOME' not in os.environ:
import pwd
userhome = pwd.getpwuid(os.getuid()).pw_dir
else:
userhome = os.environ['HOME']
Except, I don't think that GAE will allow pwd to be imported for security reasons. The sandbox environment doesn't look to have it included in its whitelist of allowed modules, and raises an ImportError if the modules aren't in that list.
If I attempt to just do an import pwd inside of a regular handler module, GAE won't allow it.