Related
I have looked through similar posts on SO and they seem to be specific to using Docker environments and haven't been much helpful. Ours is a little different, we do run a docker image of Airflow hosted on Azure App Service but it connects to hosted Azure Database for PostgreSQL server (version 11).
Python = 3.8
Apache Airflow = 2.1.4
SQL Alchemy = 1.3.24
Executor = Local
The environment has been setup and it works fine for most cases. However, when we run DAGs that handle large amounts of data (typically several GB), we suddenly encounter Heartbeat issues. Now, I have tried setting values in Airflow Config for Keep Alives through sql_alchemy_connect_args variable, and also changing the variables web_server_master_timeout and web_server_worker_timeout to a higher value to no avail.
The ERROR:
{base_job.py:222} ERROR - LocalTaskJob heartbeat got an exception
Traceback (most recent call last):
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 2336, in _wrap_pool_connect
return fn()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 364, in connect
return _ConnectionFairy._checkout(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 495, in checkout
rec = pool._do_get()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py”, line 241, in _do_get
return self._create_connection()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 309, in _create_connection
return _ConnectionRecord(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 440, in __init__
self.__connect(first_connect_check=True)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 661, in __connect
pool.logger.debug(“Error on connect(): %s”, e)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py”, line 68, in __exit__
compat.raise_(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py”, line 182, in raise_
raise exception
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 656, in __connect
connection = pool._invoke_creator(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py”, line 114, in connect
return dialect.connect(*cargs, **cparams)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py”, line 508, in connect
return self.dbapi.connect(*cargs, **cparams)
File “/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py”, line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not translate host name “<address>” to address: Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/usr/local/lib/python3.8/site-packages/airflow/jobs/base_job.py”, line 194, in heartbeat
session.merge(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py”, line 2166, in merge
return self._merge(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py”, line 2244, in _merge
merged = self.query(mapper.class_).get(key[1])
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 1018, in get
return self._get_impl(ident, loading.load_on_pk_identity)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 1135, in _get_impl
return db_load_fn(self, primary_key_identity)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py”, line 286, in load_on_pk_identity
return q.one()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 3490, in one
ret = self.one_or_none()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 3459, in one_or_none
ret = list(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 3535, in __iter__
return self._execute_and_instances(context)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 3556, in _execute_and_instances
conn = self._get_bind_args(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 3571, in _get_bind_args
return fn(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py”, line 3550, in _connection_from_session
conn = self.session.connection(**kw)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py”, line 1142, in connection
return self._connection_for_bind(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py”, line 1150, in _connection_for_bind
return self.transaction._connection_for_bind(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py”, line 433, in _connection_for_bind
conn = bind._contextual_connect()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 2339, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 1583, in _handle_dbapi_exception_noconnection
util.raise_(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py”, line 182, in raise_
raise exception
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py”, line 2336, in _wrap_pool_connect
return fn()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 364, in connect
return _ConnectionFairy._checkout(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 495, in checkout
rec = pool._do_get()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py”, line 241, in _do_get
return self._create_connection()
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 309, in _create_connection
return _ConnectionRecord(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 440, in __init__
self.__connect(first_connect_check=True)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 661, in __connect
pool.logger.debug(“Error on connect(): %s”, e)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py”, line 68, in __exit__
compat.raise_(
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py”, line 182, in raise_
raise exception
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py”, line 656, in __connect
connection = pool._invoke_creator(self)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py”, line 114, in connect
return dialect.connect(*cargs, **cparams)
File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py”, line 508, in connect
return self.dbapi.connect(*cargs, **cparams)
File “/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py”, line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name “<address>” to address: Temporary failure in name resolution
(Background on this error at: http://sqlalche.me/e/13/e3q8)
Could someone throw some light to help me navigate through this issue? I am at my wits end and I am not sure, if I am treading in the right direction debugging this.
I had the same issue with my deployment of airflow on a Kubernetes cluster. This is apparently due to a high number of simultaneous connections to the database.
I fixed it by enabling the pgBouncer as it is recommended in the official production guide.
# PgBouncer settings
pgbouncer:
# Enable PgBouncer
enabled: true
Trying to login to my app, I am getting this error when I am using a MySQL DB hosted remotely but not when I am connecting to my locally hosted XAMPP one.
I've read a couple entries on my question but they mostly mention cursors that aren't explicitly closed, which I don't need to since I am using with in all my raw queries.
I am using django-allauth for my authentication.
Traceback:
Environment:
Request Method: GET
Request URL: http://localhost:8000/accounts/login/?next=/
Django Version: 3.1.2
Python Version: 3.8.6
Installed Applications:
['ewhale_app',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.sites',
'allauth',
'allauth.account',
'users.apps.UsersConfig',
'pages.apps.PagesConfig']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback (most recent call last):
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\backends\mysql\base.py", line 73, in execute
return self.cursor.execute(query, args)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\MySQLdb\cursors.py", line 206, in execute
res = self._query(query)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\MySQLdb\cursors.py", line 319, in _query
db.query(q)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\MySQLdb\connections.py", line 259, in query
_mysql.connection.query(self, query)
The above exception ((0, '')) was the direct cause of the following exception:
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\core\handlers\exception.py", line 47, in inner
response = get_response(request)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\core\handlers\base.py", line 179, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\views\generic\base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\utils\decorators.py", line 43, in _wrapper
return bound_method(*args, **kwargs)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\views\decorators\debug.py", line 89, in sensitive_post_parameters_wrapper
return view(request, *args, **kwargs)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\allauth\account\views.py", line 138, in dispatch
return super(LoginView, self).dispatch(request, *args, **kwargs)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\allauth\account\views.py", line 78, in dispatch
response = super(RedirectAuthenticatedUserMixin,
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\views\generic\base.py", line 98, in dispatch
return handler(request, *args, **kwargs)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\allauth\account\views.py", line 94, in get
response = super(AjaxCapableProcessFormViewMixin, self).get(
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\views\generic\edit.py", line 133, in get
return self.render_to_response(self.get_context_data())
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\allauth\account\views.py", line 169, in get_context_data
site = get_current_site(self.request)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\contrib\sites\shortcuts.py", line 13, in get_current_site
return Site.objects.get_current(request)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\contrib\sites\models.py", line 58, in get_current
return self._get_site_by_id(site_id)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\contrib\sites\models.py", line 30, in _get_site_by_id
site = self.get(pk=site_id)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\models\manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\models\query.py", line 425, in get
num = len(clone)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\models\query.py", line 269, in __len__
self._fetch_all()
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\models\query.py", line 1308, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\models\query.py", line 53, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\models\sql\compiler.py", line 1156, in execute_sql
cursor.execute(sql, params)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\backends\utils.py", line 98, in execute
return super().execute(sql, params)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\backends\utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\backends\mysql\base.py", line 73, in execute
return self.cursor.execute(query, args)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\MySQLdb\cursors.py", line 206, in execute
res = self._query(query)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\MySQLdb\cursors.py", line 319, in _query
db.query(q)
File "C:\Users\Ej\AppData\Local\Programs\Python\Python38\lib\site-packages\MySQLdb\connections.py", line 259, in query
_mysql.connection.query(self, query)
Exception Type: InterfaceError at /accounts/login/
Exception Value: (0, '')
Would appreciate feedback on where to check the issue.
Avoided this issue by recreating a new DB and migrating.
I used Djano to build a simple restful API server. SSL has been introduced to the communication between the server and the remote database to enforce security. When the API server tried to write to the database, I got the following error:
File "/app/anaconda/lib/python2.7/site-packages/django/core/handlers/exception.py", line 39, in inner
response = get_response(request)
File "/app/anaconda/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/app/anaconda/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/app/anaconda/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/app/anaconda/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/app/anaconda/lib/python2.7/site-packages/rest_framework/views.py", line 474, in dispatch
response = self.handle_exception(exc)
File "/app/anaconda/lib/python2.7/site-packages/rest_framework/views.py", line 434, in handle_exception
self.raise_uncaught_exception(exc)
File "/app/anaconda/lib/python2.7/site-packages/rest_framework/views.py", line 471, in dispatch
response = handler(request, *args, **kwargs)
File "/app/anaconda/lib/python2.7/site-packages/rest_framework/decorators.py", line 52, in handler
return func(*args, **kwargs)
File "/app/mobile/mpq_backup_response/mpq_backup_response/backup_response/views.py", line 39, in sms
q.save()
File "/app/anaconda/lib/python2.7/site-packages/django/db/models/base.py", line 796, in save
force_update=force_update, update_fields=update_fields)
File "/app/anaconda/lib/python2.7/site-packages/django/db/models/base.py", line 821, in save_base
with transaction.atomic(using=using, savepoint=False):
File "/app/anaconda/lib/python2.7/site-packages/django/db/transaction.py", line 158, in __enter__
if not connection.get_autocommit():
File "/app/anaconda/lib/python2.7/site-packages/django/db/backends/base/base.py", line 365, in get_autocommit
self.ensure_connection()
File "/app/anaconda/lib/python2.7/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
self.connect()
File "/app/anaconda/lib/python2.7/site-packages/django/db/backends/base/base.py", line 171, in connect
self.connection = self.get_new_connection(conn_params)
File "/app/anaconda/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 265, in get_new_connection
conn = Database.connect(**conn_params)
File "/app/anaconda/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
return Connection(*args, **kwargs)
File "/app/anaconda/lib/python2.7/site-packages/pymysql/connections.py", line 688, in __init__
self.connect()
File "/app/anaconda/lib/python2.7/site-packages/pymysql/connections.py", line 906, in connect
self._request_authentication()
File "/app/anaconda/lib/python2.7/site-packages/pymysql/connections.py", line 1086, in _request_authentication
self._sock = self.ctx.wrap_socket(self._sock, server_hostname=self.host)
File "/app/anaconda/lib/python2.7/ssl.py", line 352, in wrap_socket
_context=self)
TypeError: __init__() got an unexpected keyword argument 'server_hostname'
However, if I went into Djangon's shell interactive environment and wrote directly into the database, everything went smoothly. I don't why this problem happens.
This issue seems to exist in OS X with python 2.7.10 and according to the link below, you will most likely run into this with CentOS 7 and Python ver 2.7.9 or above.
https://github.com/gevent/gevent/issues/477
Updating gevent to 1.2.1 seems to solve the problem.
You could run "pip install --upgrade gevent" on mac to fix this.
Newbie. Have googled, and seeing nothing.
Trying to get into admin/ and it says:
/usr/lib/python2.6/site-packages/django/db/backends/sqlite3/base.py in get_new_connection, line 346
[It is trying to use SQLlite as database].
But I have MySQL as the database--and the other functions in regards to using MySQL work correctly. (it updates MySQL database when adding new apps).
I even manually checked MySQL database, and it has the superuser that I added!
Anyone have any ideas?
Environment:
full stack trace:
Request Method: POST Request URL: ommitted/
Django Version: 1.6.2 Python Version: 2.6.6 Installed Applications: ('django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles') Installed Middleware: ('django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware')
Traceback: File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response
114. response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/lib/python2.6/site-packages/django/contrib/admin/sites.py" in wrapper
215. return self.admin_view(view, cacheable)(*args, **kwargs) File "/usr/lib/python2.6/site-packages/django/utils/decorators.py" in
_wrapped_view
99. response = view_func(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/django/views/decorators/cache.py" in
_wrapped_view_func
52. response = view_func(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/django/contrib/admin/sites.py" in inner
197. return self.login(request) File "/usr/lib/python2.6/site-packages/django/views/decorators/cache.py" in
_wrapped_view_func
52. response = view_func(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/django/contrib/admin/sites.py" in login
330. return login(request, **defaults) File "/usr/lib/python2.6/site-packages/django/views/decorators/debug.py" in sensitive_post_parameters_wrapper
75. return view(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/django/utils/decorators.py" in
_wrapped_view
99. response = view_func(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/django/views/decorators/cache.py" in
_wrapped_view_func
52. response = view_func(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/django/contrib/auth/views.py" in login
36. if form.is_valid(): File "/usr/lib/python2.6/site-packages/django/forms/forms.py" in is_valid
129. return self.is_bound and not bool(self.errors) File "/usr/lib/python2.6/site-packages/django/forms/forms.py" in errors
121. self.full_clean() File "/usr/lib/python2.6/site-packages/django/forms/forms.py" in full_clean
274. self._clean_form() File "/usr/lib/python2.6/site-packages/django/forms/forms.py" in
_clean_form
300. self.cleaned_data = self.clean() File "/usr/lib/python2.6/site-packages/django/contrib/admin/forms.py" in clean
28. self.user_cache = authenticate(username=username, password=password) File "/usr/lib/python2.6/site-packages/django/contrib/auth/__init__.py" in authenticate
49. user = backend.authenticate(**credentials) File "/usr/lib/python2.6/site-packages/django/contrib/auth/backends.py" in authenticate
16. user = UserModel._default_manager.get_by_natural_key(username) File "/usr/lib/python2.6/site-packages/django/contrib/auth/models.py" in get_by_natural_key
167. return self.get(**{self.model.USERNAME_FIELD: username}) File "/usr/lib/python2.6/site-packages/django/db/models/manager.py" in get
151. return self.get_queryset().get(*args, **kwargs) File "/usr/lib/python2.6/site-packages/django/db/models/query.py" in get
301. num = len(clone) File "/usr/lib/python2.6/site-packages/django/db/models/query.py" in
__len__
77. self._fetch_all() File "/usr/lib/python2.6/site-packages/django/db/models/query.py" in
_fetch_all
854. self._result_cache = list(self.iterator()) File "/usr/lib/python2.6/site-packages/django/db/models/query.py" in iterator
220. for row in compiler.results_iter(): File "/usr/lib/python2.6/site-packages/django/db/models/sql/compiler.py" in results_iter
709. for rows in self.execute_sql(MULTI): File "/usr/lib/python2.6/site-packages/django/db/models/sql/compiler.py" in execute_sql
781. cursor = self.connection.cursor() File "/usr/lib/python2.6/site-packages/django/db/backends/__init__.py" in cursor
157. cursor = self.make_debug_cursor(self._cursor()) File "/usr/lib/python2.6/site-packages/django/db/backends/__init__.py" in _cursor
129. self.ensure_connection() File "/usr/lib/python2.6/site-packages/django/db/backends/__init__.py" in ensure_connection
124. self.connect() File "/usr/lib/python2.6/site-packages/django/db/utils.py" in __exit__
99. six.reraise(dj_exc_type, dj_exc_value, traceback) File "/usr/lib/python2.6/site-packages/django/db/backends/__init__.py" in ensure_connection
124. self.connect() File "/usr/lib/python2.6/site-packages/django/db/backends/__init__.py" in connect
112. self.connection = self.get_new_connection(conn_params) File "/usr/lib/python2.6/site-packages/django/db/backends/sqlite3/base.py" in get_new_connection
346. conn = Database.connect(**conn_params)
Exception Type: OperationalError at /admin/ Exception Value: unable to open database file
settings.py database settings:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'pymail',
'USER': 'pymail',
'PASSWORD': 'removed',
'HOST': 'removed',
'PORT': '3306',
}
}
I am using SQLAlchemy 0.8 with Google App Engine and Cloud SQL. I would like to do a full integration test locally before deploying, but I cannot get SQLAlchemy to connect to my development MySQL server using the mysql+gaerdbms dialect
create_engine(
'mysql+gaerdbms:///test?instance=homingbox:instance1'
)
I start my app engine from the command line as follows
python dev_appserver.py --mysql_user=username --mysql_password=root ../Source/HomingBox/gae
Whenever I try to hit the database I get this lovely stack trace
ERROR 2013-04-22 21:35:39,987 webapp2.py:1528] (ImportError) None None
Traceback (most recent call last):
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/home/leon/Development/google_appengine/lib/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/home/leon/Development/Source/HomingBox/gae/rest.py", line 19, in get_list
self.response.write(json.dumps(application.get_list()))
File "/home/leon/Development/Source/HomingBox/gae/dal.py", line 98, in get_list
return query.all()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/query.py", line 2140, in all
return list(self)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/query.py", line 2252, in __iter__
return self._execute_and_instances(context)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/query.py", line 2265, in _execute_and_instances
close_with_result=True)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/query.py", line 2256, in _connection_from_session
**kw)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/session.py", line 797, in connection
close_with_result=close_with_result)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/session.py", line 801, in _connection_for_bind
return self.transaction._connection_for_bind(engine)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/orm/session.py", line 297, in _connection_for_bind
conn = bind.contextual_connect()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/engine/base.py", line 1669, in contextual_connect
self.pool.connect(),
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 272, in connect
return _ConnectionFairy(self).checkout()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 425, in __init__
rec = self._connection_record = pool._do_get()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 855, in _do_get
return self._create_connection()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 225, in _create_connection
return _ConnectionRecord(self)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 318, in __init__
self.connection = self.__connect()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/pool.py", line 368, in __connect
connection = self.__pool._creator()
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/engine/strategies.py", line 80, in connect
return dialect.connect(*cargs, **cparams)
File "/home/leon/Development/Source/HomingBox/gae/sqlalchemy/engine/default.py", line 279, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 183, in __init__
super(GoogleApiConnection, self).__init__(*args, **kwargs)
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms.py", line 810, in __init__
self.OpenConnection()
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms.py", line 832, in OpenConnection
self.SetupClient()
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 193, in SetupClient
self._client = RdbmsGoogleApiClient(**kwargs)
File "/home/leon/Development/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 106, in __init__
rdbms.OAUTH_CREDENTIALS_PATH)
File "/usr/lib/python2.7/posixpath.py", line 268, in expanduser
import pwd
File "/home/leon/Development/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 827, in load_module
raise ImportError
Any help is much appreciated
Here's my hypothesis, based on the stack trace -- when the application can't find your HOME directory, it attempts to import pwd and get the home directory information that way. The application is trying to call this bit of code in posixpath.py:
if 'HOME' not in os.environ:
import pwd
userhome = pwd.getpwuid(os.getuid()).pw_dir
else:
userhome = os.environ['HOME']
Except, I don't think that GAE will allow pwd to be imported for security reasons. The sandbox environment doesn't look to have it included in its whitelist of allowed modules, and raises an ImportError if the modules aren't in that list.
If I attempt to just do an import pwd inside of a regular handler module, GAE won't allow it.