Executing new database queries while looping through a query result - mysql

Question

I have a relatively heavy MySQL select query that returns ca 150 MB of data in 25.000 rows. Looping through the whole dataset and performing the required operations, takes about 45 minutes.
For a few of the records (less than 10) I need to perform another lookup in the same database.
If I write my Python 3 code like this:
con = mysql.connection.connect( host='...', user='...', password='...', database='...' )
cursor = con.cursor( dictionary=True, buffered=True )
cursor.execute('SELECT ...')
for row in cursor:
# processing data ...
if row['...'] == ...:
cursor2 = con.cursor()
cursor2.execute('SELECT ...')
some_var = cursor2.fetch_one()[0]
# more processing...
Then, at the second execution of cursor2, I get:
mysql.connector.errors.OperationalError: MySQL Connection not available.
If I initialize cursor2 before the loop:
con = mysql.connection.connect( host='...', user='...', password='...', database='...' )
cursor = con.cursor( dictionary=True, buffered=True )
cursor2 = con.cursor()
cursor.execute('SELECT ...')
for row in cursor:
# processing data ...
if row['...'] == ...:
cursor2.execute('SELECT ...')
some_var = cursor2.fetch_one()[0]
# more processing...
Then, at the second execution I get:
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
I'm not really sure if these errors are caused by time-outs, or if it is caused by my wrong handling of multiple cursors.
If it helps, these are the timeout settings of my database:
connect_timeout 10
deadlock_timeout_long 50000000
deadlock_timeout_short 10000
delayed_insert_timeout 300
idle_readonly_transaction_timeout 0
idle_transaction_timeout 0
idle_write_transaction_timeout 0
innodb_flush_log_at_timeout 1
innodb_lock_wait_timeout 50
innodb_rollback_on_timeout OFF
interactive_timeout 28800
lock_wait_timeout 86400
net_read_timeout 30
net_write_timeout 60
rpl_semi_sync_master_timeout 10000
rpl_semi_sync_slave_kill_conn_timeout 5
slave_net_timeout 60
thread_pool_idle_timeout 60
wait_timeout 420
How should I write this code instead? This is only my second Python script, so please formulate any help at a beginners level. 😉

Answer 1

It turns out connections are behaving a bit different from what I was expecting. The comment from #Mike67 pointed in the right direction, I just missing some of the details. The solution is to create and destroy a new connection every time a new query is needed.
con = mysql.connection.connect( host='...', user='...', password='...', database='...' )
cursor = con.cursor( dictionary=True, buffered=True )
cursor.execute('SELECT ...')
for row in cursor:
# processing data ...
if row['...'] == ...:
con2 = mysql.connection.connect( host='...', user='...', password='...', database='...' )
cursor2 = con2.cursor()
cursor2.execute('SELECT ...')
some_var = cursor2.fetch_one()[0]
cursor2.close()
con2.close()
# more processing...
I have a feeling there must be a more efficient way to do this, but this works.