SQLAlchemy hangs when external db connection fails - sqlalchemy

Question

In my application, I use an internal db and an external db where I use the external db to load some details from the main/global database.
From time to time when I try to query some data or commit a transaction, I get an error sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid transaction is rolled back which causes whole application to break down and it's not fixed until I restart it. This mostly happen when the external database is being restarted. We lose connection, the transaction gets invalid and it hangs.
In the FastAPI app, I use SQLAlchemy. I set the external DB as follows (with some redundant details omitted):
class _ExternalDB(DBSetup):
def __init__(self):
self._database = Database(self.url, ssl=ctx)
self._engine = create_engine(self.url, echo=True, echo_pool='debug')
self._session = scoped_session(
sessionmaker(autocommit=False, autoflush=False, bind=self._engine, class_=SafeSession)
)
where DBSetup is a base class which contains url, database and session. I wrote SafeSession class to try and solve the issue I am having:
class SafeSession(Session):
def commit(self):
try:
return super().commit()
except Exception as e:
logger.error("Could not commit the SQL operation!", e)
self.rollback()
raise e
def query(self, *entities, **kwargs):
try:
return super().query(*entities, **kwargs)
except Exception as e:
logger.error("Could not complete the transaction!", e)
self.rollback()
raise e
An example query is as follows:
external_db_setup.session.query(User).get(user_id)
My goal was to roll the invalid transaction back whenever it happens, but the application never gets to the except block because I have never seen any logs for "Could not complete the transaction". Am I missing something?
Full error log:
File "/***/repositories/user.py", line 8, in get_user_by_id
return external_db_setup.session.query(User).get(user_id)
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1004, in get
return self._get_impl(ident, loading.load_on_pk_identity)
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1119, in _get_impl
return db_load_fn(self, primary_key_identity)
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 284, in load_on_pk_identity
return q.one()
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3358, in one
ret = self.one_or_none()
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3327, in one_or_none
ret = list(self)
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3403, in __iter__
return self._execute_and_instances(context)
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3428, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/***/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 984, in execute
return meth(self, multiparams, params)
File "/***/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/***/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1097, in _execute_clauseelement
ret = self._execute_context(
File "/***/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1177, in _execute_context
self._handle_dbapi_exception(
File "/***/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _handle_dbapi_exception
util.raise_(
File "/***/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/***/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1173, in _execute_context
conn = self._revalidate_connection()
File "/***/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 456, in _revalidate_connection
raise exc.InvalidRequestError(
graphql.error.located_error.GraphQLLocatedError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back