Airflow heartbeat FATAL: sorry, too many clients already - sqlalchemy

Has anyone encountered "sorry, too many clients already" error on heartbeat phase while running a subdag?
[2017-11-22 08:21:23,747] {jobs.py:2136} ERROR - Exception while
trying to heartbeat! Sleeping for 5.0s Traceback (most recent call
last): File
"/usr/local/lib/python2.7/site-packages/airflow/jobs.py", line 2131,
in _execute
self.heartbeat() File "/usr/local/lib/python2.7/site-packages/airflow/jobs.py", line 180, in
heartbeat
job = session.query(BaseJob).filter(BaseJob.id == self.id).first() File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py",
line 2755, in first
ret = list(self[0:1]) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line
2547, in getitem
return list(res) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line
2855, in iter
return self._execute_and_instances(context) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line
2876, in _execute_and_instances
close_with_result=True) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line
2885, in _get_bind_args
**kw File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line
2867, in _connection_from_session
conn = self.session.connection(**kw) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 998, in connection
execution_options=execution_options) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 1003, in _connection_for_bind
engine, execution_options) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 403, in _connection_for_bind
conn = bind.contextual_connect() File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 2112, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None), File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 2151, in _wrap_pool_connect
e, dialect, self) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1465, in _handle_dbapi_exception_noconnection
exc_info File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
return fn() File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 387,
in connect
return _ConnectionFairy._checkout(self) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 766,
in _checkout
fairy = _ConnectionRecord.checkout(pool) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 516,
in checkout
rec = pool._do_get() File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line
1229, in _do_get
return self._create_connection() File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 333,
in _create_connection
return _ConnectionRecord(self) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 461,
in init
self.connect(first_connect_check=True) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 651,
in __connect
connection = pool._invoke_creator(self) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
return dialect.connect(*cargs, **cparams) File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
return self.dbapi.connect(*cargs, **cparams) File "/usr/local/lib/python2.7/site-packages/psycopg2/__init.py", line
130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync) OperationalError: (psycopg2.OperationalError) FATAL: sorry, too many clients already
from airflow.cfg:
on airflow.cfg:
sql_alchemy_pool_size = 100
sql_alchemy_pool_recycle = 3600
Is there a known solution to this problem?
I'm using airflow 1.8.2 with LocalExecutor.

Related

Connection refused when switching TPU version

How can I switch TPU version for TPU VM architechture?
When attempting to switch software version for TPU(TPU VM architechture switching from tpu-vm-tf-2.6.0-pod to tpu-vm-base) using instructions found here, I get Connection Refused exception with traceback:
Traceback (most recent call last):
File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/lib/python3.8/http/client.py", line 1256, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 951, in send
self.connect()
File "/usr/lib/python3.8/http/client.py", line 922, in connect
self.sock = self._create_connection(
File "/usr/lib/python3.8/socket.py", line 808, in create_connection
raise err
File "/usr/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "switch.py", line 20, in <module>
c.configure_tpu_version(args.target_version, restart_type="ifNeeded")
File "/usr/local/lib/python3.8/dist-packages/cloud_tpu_client/client.py", line 391, in configure_tpu_version
for result in results:
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
yield fs.pop().result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.8/dist-packages/cloud_tpu_client/client.py", line 375, in configure_worker
request.urlopen(req)
File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.8/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 1383, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/lib/python3.8/urllib/request.py", line 1357, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
The command is run on cloud TPU VM with cloud-tpu-client version 0.10
When running the same command from my PC I get Connection timed out after a long pause with traceback:
Traceback (most recent call last):
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/urllib/request.py", line 1350, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/http/client.py", line 1281, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/http/client.py", line 1327, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/http/client.py", line 1276, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/http/client.py", line 1036, in _send_output
self.send(msg)
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/http/client.py", line 976, in send
self.connect()
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/http/client.py", line 948, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "tpu_version.py", line 19, in <module>
c.configure_tpu_version(args.target_version, restart_type="ifNeeded")
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/site-packages/cloud_tpu_client/client.py", line 392, in configure_tpu_version
for result in results:
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/concurrent/futures/_base.py", line 598, in result_iterator
yield fs.pop().result()
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/concurrent/futures/_base.py", line 435, in result
return self.__get_result()
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/site-packages/cloud_tpu_client/client.py", line 376, in configure_worker
request.urlopen(req)
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/urllib/request.py", line 1378, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/home/nevus/anaconda3/envs/imageGen/lib/python3.7/urllib/request.py", line 1352, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>
This feature is not supported by the TPU VM architecture.
Best way to change the TPU VM's version would be to delete it and recreate a new one with the desired version.

psycopg2.OperationalError: could not translate host name "<address>" to address: Temporary failure in name resolution

I have looked through similar posts on SO and they seem to be specific to using Docker environments and haven't been much helpful. Ours is a little different, we do run a docker image of Airflow hosted on Azure App Service but it connects to hosted Azure Database for PostgreSQL server (version 11).
Python = 3.8
Apache Airflow = 2.1.4
SQL Alchemy = 1.3.24
Executor = Local
The environment has been setup and it works fine for most cases. However, when we run DAGs that handle large amounts of data (typically several GB), we suddenly encounter Heartbeat issues. Now, I have tried setting values in Airflow Config for Keep Alives through sql_alchemy_connect_args variable, and also changing the variables web_server_master_timeout and web_server_worker_timeout to a higher value to no avail.
The ERROR:
{base_job.py:222} ERROR - LocalTaskJob heartbeat got an exception
Traceback (most recent call last):
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 2336, in _wrap_pool_connect
return fn()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 364, in connect
return _ConnectionFairy._checkout(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 495, in checkout
rec = pool._do_get()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py”, line 241, in _do_get
return self._create_connection()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 309, in _create_connection
return _ConnectionRecord(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 440, in __init__
self.__connect(first_connect_check=True)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 661, in __connect
pool.logger.debug(“Error on connect(): %s”, e)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py”, line 68, in __exit__
compat.raise_(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py”, line 182, in raise_
raise exception
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 656, in __connect
connection = pool._invoke_creator(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py”, line 114, in connect
return dialect.connect(*cargs, **cparams)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py”, line 508, in connect
return self.dbapi.connect(*cargs, **cparams)
File “/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py”, line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not translate host name “<address>” to address: Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/usr/local/lib/python3.8/site-packages/airflow/jobs/base_job.py”, line 194, in heartbeat
session.merge(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py”, line 2166, in merge
return self._merge(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py”, line 2244, in _merge
merged = self.query(mapper.class_).get(key[1])
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 1018, in get
return self._get_impl(ident, loading.load_on_pk_identity)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 1135, in _get_impl
return db_load_fn(self, primary_key_identity)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py”, line 286, in load_on_pk_identity
return q.one()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 3490, in one
ret = self.one_or_none()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 3459, in one_or_none
ret = list(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 3535, in __iter__
return self._execute_and_instances(context)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 3556, in _execute_and_instances
conn = self._get_bind_args(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 3571, in _get_bind_args
return fn(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 3550, in _connection_from_session
conn = self.session.connection(**kw)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py”, line 1142, in connection
return self._connection_for_bind(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py”, line 1150, in _connection_for_bind
return self.transaction._connection_for_bind(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py”, line 433, in _connection_for_bind
conn = bind._contextual_connect()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 2339, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 1583, in _handle_dbapi_exception_noconnection
util.raise_(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py”, line 182, in raise_
raise exception
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 2336, in _wrap_pool_connect
return fn()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 364, in connect
return _ConnectionFairy._checkout(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 495, in checkout
rec = pool._do_get()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py”, line 241, in _do_get
return self._create_connection()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 309, in _create_connection
return _ConnectionRecord(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 440, in __init__
self.__connect(first_connect_check=True)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 661, in __connect
pool.logger.debug(“Error on connect(): %s”, e)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py”, line 68, in __exit__
compat.raise_(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py”, line 182, in raise_
raise exception
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 656, in __connect
connection = pool._invoke_creator(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py”, line 114, in connect
return dialect.connect(*cargs, **cparams)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py”, line 508, in connect
return self.dbapi.connect(*cargs, **cparams)
File “/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py”, line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name “<address>” to address: Temporary failure in name resolution
(Background on this error at: http://sqlalche.me/e/13/e3q8)
Could someone throw some light to help me navigate through this issue? I am at my wits end and I am not sure, if I am treading in the right direction debugging this.
I had the same issue with my deployment of airflow on a Kubernetes cluster. This is apparently due to a high number of simultaneous connections to the database.
I fixed it by enabling the pgBouncer as it is recommended in the official production guide.
# PgBouncer settings
pgbouncer:
# Enable PgBouncer
enabled: true

Is there a solution in fixing 'MySQL connection not available' for very second request in Flask application?

I am running a Flask application on local and production server. I have no issues with the local, I am facing 'MySQL connection not available' for every second database request on production server. After reloading or after performing rollback operation, it is getting executed, but the issue repeats. I tried major solutions available on the internet by changing, pool_recycle, 'wait_timeout' and other timeouts to same value of 1600, but didn't work. I finally destroyed the application and reinstalled everything on the server but the issue is still on. Please help in this, thank you.
My production MySQL config:
class ProductionConfig(Config):
SECRET_KEY = 'secret_key'
DEBUG=False
SQLALCHEMY_DATABASE_URI = "mysql+pymysql://{username}:
{password}#{hostname}/{databasename}".format(
username="myusername",
password="password",
hostname="myhostname",
databasename="mydatabase",
)
SQLALCHEMY_POOL_RECYCLE = 299
SQLALCHEMY_TRACK_MODIFICATIONS = False
Error Message:
Exception on /add_questions/1/2 [POST]
Traceback (most recent call last):
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 713, in _write_bytes
self._sock.sendall(data)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
cursor, statement, parameters, context
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
cursor.execute(statement, parameters)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 516, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 771, in _execute_command
self._write_bytes(packet)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 718, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask_login/utils.py", line 270, in decorated_view
elif not current_user.is_authenticated:
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/werkzeug/local.py", line 347, in __getattr__
return getattr(self._get_current_object(), name)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/werkzeug/local.py", line 306, in _get_current_object
return self.__local()
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask_login/utils.py", line 26, in <lambda>
current_user = LocalProxy(lambda: _get_user())
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask_login/utils.py", line 346, in _get_user
current_app.login_manager._load_user()
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/flask_login/login_manager.py", line 318, in _load_user
user = self._user_callback(user_id)
File "/home/ulznrcvr/jaihindpro/application/auth/views.py", line 26, in load_user
return User.query.get(int(user_id))
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 1004, in get
return self._get_impl(ident, loading.load_on_pk_identity)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 1121, in _get_impl
return db_load_fn(self, primary_key_identity)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 287, in load_on_pk_identity
return q.one()
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3360, in one
ret = self.one_or_none()
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3329, in one_or_none
ret = list(self)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3405, in __iter__
return self._execute_and_instances(context)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3430, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 984, in execute
return meth(self, multiparams, params)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1103, in _execute_clauseelement
distilled_params,
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1288, in _execute_context
e, statement, parameters, cursor, context
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1482, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
cursor, statement, parameters, context
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
cursor.execute(statement, parameters)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 516, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 771, in _execute_command
self._write_bytes(packet)
File "/home/ulznrcvr/virtualenv/jaihindpro/3.6/lib/python3.6/site-packages/pymysql/connections.py", line 718, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
[SQL: SELECT user.id AS user_id, user.student_id AS user_student_id, user.role_id AS user_role_id, user.course_id AS user_course_id, user.sub_course_id AS user_sub_course_id, user.batch_id AS user_batch_id, user.full_name AS user_full_name, user.email AS user_email, user.phone_number AS user_phone_number, user.password AS user_password, user.active AS user_active, user.confirmed AS user_confirmed, user.confirmed_date AS user_confirmed_date, user.subscribed AS user_subscribed, user.start_date AS user_start_date, user.end_date AS user_end_date, user.re_url AS user_re_url, user.online AS user_online
FROM user
WHERE user.id = %(param_1)s]
[parameters: {'param_1': 2}]
(Background on this error at: http://sqlalche.me/e/e3q8)
I figured it out myself, I flagged "pool_pre_ping" attribute to "True" in engine settings.
SQLALCHEMY_ENGINE_OPTIONS = {
"pool_pre_ping": True,
"pool_recycle": 300,
}
Ref: https://medium.com/#heyjcmc/controlling-the-flask-sqlalchemy-engine-a0f8fae15b47

Airflow initdb with mysql as backend is not working

when I initialize airflow using mysql as backend I see below error
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py",
line 453, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.7/site-packages/MySQLdb/init.py", line 80,
in Connect
from MySQLdb.connections import Connection
File "/usr/local/lib/python3.7/site-packages/MySQLdb/connections.py", line
36
raise errorclass, errorvalue
^
SyntaxError: invalid syntax
I tried re-installing mysql-python and pymysql still its same
airflow initdb
/usr/local/lib/python3.7/site-packages/airflow/configuration.py:627: DeprecationWarning: You have two airflow.cfg files: /home/airflow/airflow/airflow.cfg and /home/airflow/airflow.cfg. Airflow used to look at ~/airflow/airflow.cfg, even when AIRFLOW_HOME was set to a different value. Airflow will now only read /home/airflow/airflow.cfg, and you should remove the other file
category=DeprecationWarning,
[2019-09-24 12:10:32,310] {settings.py:213} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=15403
[2019-09-24 12:10:32,508] {__init__.py:51} INFO - Using executor LocalExecutor
DB: mysql://airdba:***#host/airflow_main
[2019-09-24 12:10:32,820] {db.py:369} INFO - Creating tables
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 32, in <module>
args.func(args)
File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1102, in initdb
db.initdb(settings.RBAC)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 106, in initdb
upgradedb()
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 377, in upgradedb
command.upgrade(config, 'heads')
File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 298, in upgrade
script.run_env()
File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 489, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.7/site-packages/alembic/util/compat.py", line 173, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.7/site-packages/airflow/migrations/env.py", line 92, in <module>
run_migrations_online()
File "/usr/local/lib/python3.7/site-packages/airflow/migrations/env.py", line 78, in run_migrations_online
with connectable.connect() as connection:
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2206, in connect
return self._connection_cls(self, **kwargs)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 103, in __init__
else engine.raw_connection()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2306, in raw_connection
self.pool.unique_connection, _connection
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2275, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
rec = pool._do_get()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
self._dec_overflow()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
return self._create_connection()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 453, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.7/site-packages/MySQLdb/__init__.py", line 80, in Connect
from MySQLdb.connections import Connection
File "/usr/local/lib/python3.7/site-packages/MySQLdb/connections.py", line 36
raise errorclass, errorvalue
^
SyntaxError: invalid syntax
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 453, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.7/site-packages/MySQLdb/__init__.py", line 80, in Connect
from MySQLdb.connections import Connection
File "/usr/local/lib/python3.7/site-packages/MySQLdb/connections.py", line 36
raise errorclass, errorvalue
^
SyntaxError: invalid syntax
removed mysql-python module and forced airflow to use pymysql.
changed connection string in airflow.cfg
sql_alchemy_conn = mysql+pymysql://user:password#server/dbname

SQLAlchemy with Google Cloud SQL connect to development server

I am using SQLAlchemy 0.8 with Google App Engine and Cloud SQL. I would like to do a full integration test locally before deploying, but I cannot get SQLAlchemy to connect to my development MySQL server using the mysql+gaerdbms dialect
create_engine(
'mysql+gaerdbms:///test?instance=homingbox:instance1'
)
I start my app engine from the command line as follows
python dev_appserver.py --mysql_user=username --mysql_password=root ../Source/HomingBox/gae
Whenever I try to hit the database I get this lovely stack trace
ERROR 2013-04-22 21:35:39,987 webapp2.py:1528] (ImportError) None None
Traceback (most recent call last):
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/home/leon/Development/Source/HomingBox/gae/rest.py", line 19, in get_list
self.response.write(json.dumps(application.get_list()))
File "/home/leon/Development/Source/HomingBox/gae/dal.py", line 98, in get_list
return query.all()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/query.py", line 2140, in all
return list(self)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/query.py", line 2252, in __iter__
return self._execute_and_instances(context)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/query.py", line 2265, in _execute_and_instances
close_with_result=True)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/query.py", line 2256, in _connection_from_session
**kw)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/session.py", line 797, in connection
close_with_result=close_with_result)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/session.py", line 801, in _connection_for_bind
return self.transaction._connection_for_bind(engine)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/session.py", line 297, in _connection_for_bind
conn = bind.contextual_connect()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/engine/base.py", line 1669, in contextual_connect
self.pool.connect(),
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 272, in connect
return _ConnectionFairy(self).checkout()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 425, in __init__
rec = self._connection_record = pool._do_get()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 855, in _do_get
return self._create_connection()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 225, in _create_connection
return _ConnectionRecord(self)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 318, in __init__
self.connection = self.__connect()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 368, in __connect
connection = self.__pool._creator()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/engine/strategies.py", line 80, in connect
return dialect.connect(*cargs, **cparams)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/engine/default.py", line 279, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 183, in __init__
super(GoogleApiConnection, self).__init__(*args, **kwargs)
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms.py", line 810, in __init__
self.OpenConnection()
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms.py", line 832, in OpenConnection
self.SetupClient()
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 193, in SetupClient
self._client = RdbmsGoogleApiClient(**kwargs)
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 106, in __init__
rdbms.OAUTH_CREDENTIALS_PATH)
File "/usr/lib/python2.7/posixpath.py", line 268, in expanduser
import pwd
File "/home/leon/Development/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 827, in load_module
raise ImportError
Any help is much appreciated
Here's my hypothesis, based on the stack trace -- when the application can't find your HOME directory, it attempts to import pwd and get the home directory information that way. The application is trying to call this bit of code in posixpath.py:
if 'HOME' not in os.environ:
import pwd
userhome = pwd.getpwuid(os.getuid()).pw_dir
else:
userhome = os.environ['HOME']
Except, I don't think that GAE will allow pwd to be imported for security reasons. The sandbox environment doesn't look to have it included in its whitelist of allowed modules, and raises an ImportError if the modules aren't in that list.
If I attempt to just do an import pwd inside of a regular handler module, GAE won't allow it.