How to handle connection drops - mysql

Question

I'm using a connection pool and I'm clueless about what to do when the mysql server drops my client's connection due to inactivity/mysql server goes down. I'm calling the below function everytime I've to make a query:
def getDbCnx():
try:
dbConn = mysql.connector.connect(pool_name = "connectionPool", pool_size = 3, pool_reset_session=True, **dbConfig)
except mysql.connector.Error as err:
if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
print("Something is wrong with your user name or password")
dbConn.close()
return None
elif err.errno == errorcode.ER_BAD_DB_ERROR:
print("Database does not exist")
dbConn.close()
return None
else:
print(err)
dbConn.close()
return None
else:
return dbConn
As per my current understanding, the connection pool will be initialised on the first call of this function. And after that, it will just return a free connection from the already initialised pool. Now, suppose connection pool gets initialised successfully on the first call. And after sometime, say the mysql server goes down or it drops the connection due to inactivity. What will happen, when I query after such a situation. Because I suppose the older context would have gone stale.
Basically how do I ensure that the connection pool refreshes its internal contexts everytime it loses connectivity with the mysql server.

Answer 1

When you invoke dbConn.close() The connection will be reset (and we can observe the source here: https://github.com/mysql/mysql-connector-python/.../mysql/connector/pooling.py#L118 we can expect session variables deallocated, lost uncommitted transactions, etc.). The connection is not fully closed and it can be check by printing the connection id (it should not change if it is the same connection).
Once you attempt to retrieve another connection from the pool with mysql.connector.connect(pool_name = "connectionPool") it will check the connection and if the connection can not be reconnected, a new connection will be open (with a new session id), but in the case the new connection fails an error will be raised. So, if there is a server online and the user account you are using exist in the server is almost certain you will get the connection if the pool is not exhausted and server is online, even if the server was restarted or if you have updated your server after the creation of the connection pool, and also if the server has closed the inactive session, so make sure you close the connection so it can back to the pool and can be reused.
In the below example I shutdown the server with SHUTDOWN command from the MySQL console and then restart it with mysqladmin, you can see the connection id of each connection in the pool (some connections where reused), and that the variables are deallocated due to the connection being reset when goes back to the pool.
from time import sleep
import mysql.connector
from mysql.connector import errorcode
from builtins import range
from mysql.connector import errors
dbConfig = {
'host': '127.0.0.1',
'user': 'some_user', 'password': 'some_pass',
'port': 4824,
}
def getDbCnx():
try:
dbConn = mysql.connector.connect(
pool_name = "connectionPool",
pool_size = 3,
pool_reset_session=True,
**dbConfig
)
return dbConn
except (AttributeError, errors.InterfaceError) as err:
# Errors from bad configuration. not supported options or not valid
print(f"Something is wrong with the connection pool: {err}", flush=True)
except (errors.PoolError) as err:
# Errors from bad connection pool configuration or pool exhausted
print(f"Something is wrong with the connection pool: {err}", flush=True)
except errors.OperationalError as err:
# Errors from MySQL like Lost connection 2013 2055
print(f"Something is wrong with the MySQL server: {err}", flush=True)
except errors.ProgrammingError as err:
# Errors from bad connection data
if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
print("Something is wrong with your user name or password", flush=True)
elif err.errno == errorcode.ER_BAD_DB_ERROR:
print("Database does not exist", flush=True)
except mysql.connector.Error as err:
print(f"{err}", flush=True)
print(f"err type: {type(err)}")
return None
def can_connect():
print("Getting connections...")
greetings = ["hello", "hi", "howdy", "hola"]
for n in range(4):
print(f"getting connection {n}")
cnx = getDbCnx()
if not cnx:
print("No database connection!!!")
return False
cur = cnx.cursor()
cur.execute("select connection_id()")
res = cur.fetchall()
print(f"connection id: {res}")
cur.execute('show variables like "%greeting%"')
res = cur.fetchall()
print(f"greeting?: {res}")
cur.execute(f"select #greeting")
greet = cur.fetchall()
print(f"greet: {greet}")
cur.execute(f"SET #greeting='{greetings[n]}'")
cur.execute(f"select #greeting")
greet = cur.fetchall()
print(f"greet: {greet}\n")
cur.close()
cnx.close()
print("")
return True
def pause(sleep_secs=30, count_down=29):
sleep(sleep_secs)
for s in range(count_down, 0, -1):
print(f"{s}, ", end='')
sleep(1)
print()
def test():
print("Initial test")
assert can_connect()
print("\nStop the server now...")
pause(10, 20)
print("\ntest with server stoped")
print("\ngetting connections with server shutdown should fail")
assert not can_connect()
print("\nStart the server now...")
pause()
print("\ntest if we can get connections again")
print("second test")
assert can_connect()
if __name__ == "__main__":
test()
Here is the output of the example above, even if the server was shutdown you can still retrieve connections once the server comes back online:
Initial test
Getting connections...
getting connection 0
connection id: [(9,)]
greeting?: []
greet: [(None,)]
greet: [('hello',)]
getting connection 1
connection id: [(10,)]
greeting?: []
greet: [(None,)]
greet: [('hi',)]
getting connection 2
connection id: [(11,)]
greeting?: []
greet: [(None,)]
greet: [('howdy',)]
getting connection 3
connection id: [(9,)]
greeting?: []
greet: [(None,)]
greet: [('hola',)]
Stop the server now...
20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1,
test with server stoped
getting connections with server shutdown should fail
Getting connections...
getting connection 0
Something is wrong with the connection pool: Can not reconnect to MySQL after 1 attempt(s): 2003: Can't connect to MySQL server on '127.0.0.1:4824' (10061 No connection could be made because the target machine actively refused it)
No database connection!!!
Start the server now...
29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1,
test if we can get connections again
second test
Getting connections...
getting connection 0
connection id: [(23,)]
greeting?: []
greet: [(None,)]
greet: [('hello',)]
getting connection 1
connection id: [(24,)]
greeting?: []
greet: [(None,)]
greet: [('hi',)]
getting connection 2
connection id: [(25,)]
greeting?: []
greet: [(None,)]
greet: [('howdy',)]
getting connection 3
connection id: [(23,)]
greeting?: []
greet: [(None,)]
greet: [('hola',)]
We can see that the first time we retrieve connections from the pool we have the connections ids [9, 10, 11] and the connection 9 was reused. Later when the shutdown the connection the "No database connection!!!" text is printed and after I started the server the connections ids where [23, 24, 25] and the connection with id 23 was reused. In addition the greeting variable was deallocated in the server.