Over time losing connection to mysql server during queries - mysql

I am running a discord bot using a MySQL server which are BOTH running on my UBUNTU 18.04 server. It works fine, but after a few hours I start to get an error any time I access the database.
Traceback (most recent call last):
File "/home/narnar/.local/lib/python3.7/site-packages/discord_slash/client.py", line 1352, in invoke_command
await func.invoke(ctx, **args)
File "/home/narnar/.local/lib/python3.7/site-packages/discord_slash/model.py", line 209, in invoke
return await self.func(self.cog, *args, **kwargs)
File "/home/narnar/cool-art/cogs/artlevels.py", line 120, in leaderboardCommand
cur.execute("SELECT * FROM artLevels")
File "/home/narnar/.local/lib/python3.7/site-packages/mysql/connector/cursor_cext.py", line 271, in execute
raw_as_string=self._raw_as_string)
File "/home/narnar/.local/lib/python3.7/site-packages/mysql/connector/connection_cext.py", line 522, in cmd_query
sqlstate=exc.sqlstate)
mysql.connector.errors.OperationalError: 2013 (HY000): Lost connection to MySQL server during query

I solved this issue by not having connections open for too long. My program connected to the MySQL server at the beginning of the program and used that same connection all the time. So instead of having one connection, I just opened a connection when I needed to access the database then closed it afterwards. Just make sure you don't have your connection open for too long
Don't do this:
con = mysql.connector.connect(
host=host,
user="ImNotThat",
passwd="Stupid",
database="toShowMyPasswords"
)
async def someListenerEvent():
doThing()
con.commit()
Do this:
async def someListenerEvent():
con = mysql.connector.connect(
host=host,
user="ImNotThat",
passwd="Stupid",
database="toShowMyPasswords"
)
doThing()
con.commit()
cur.close()
con.close()

Related

Using multiple gunicorn workers cause the error with status PGRES_TUPLES_OK and no message from the libpq

I have a Flask website that uses the peewee ORM. The connection is managed by FlaskDB.
When I only use 1 gunicorn worker, it works well. But as soon as I use 2 or more gunicorn workers, I start getting this error:
Traceback (most recent call last):
File "/home/user/.local/lib/python3.10/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/home/user/.local/lib/python3.10/site-packages/flask/app.py", line 1519, in full_dispatch_request
return self.finalize_request(rv)
File "/home/user/.local/lib/python3.10/site-packages/flask/app.py", line 1540, in finalize_request
response = self.process_response(response)
File "/home/user/.local/lib/python3.10/site-packages/flask/app.py", line 1888, in process_response
self.session_interface.save_session(self, ctx.session, response)
File "/home/user/project/session.py", line 113, in save_session
saved_session.save()
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 6497, in save
rows = self.update(**field_dict).where(self._pk_expr()).execute()
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 1886, in inner
return method(self, database, *args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 1957, in execute
return self._execute(database)
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 2442, in _execute
cursor = database.execute(self)
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 3112, in execute
return self.execute_sql(sql, params, commit=commit)
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 3096, in execute_sql
with __exception_wrapper__:
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 2873, in __exit__
reraise(new_type, new_type(exc_value, *exc_args), traceback)
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 183, in reraise
raise value.with_traceback(tb)
File "/home/user/.local/lib/python3.10/site-packages/peewee.py", line 3099, in execute_sql
cursor.execute(sql, params or ())
peewee.DatabaseError: error with status PGRES_TUPLES_OK and no message from the libpq
The peewee documentation states that the connection should be thread-safe, but it seems there are thread-safety issues here.
I use playhouse.pool.PooledPostgresqlDatabase if that matters.
What is the solution to this problem?
I believe this is possibly due to the connection being opened before the workers are forked. I'm not too familiar w/gunicorns worker model, but googling your error reveals similar problems w/multiprocessing. Specifically,
When a program uses multiprocessing or fork(), and an Engine object is copied to the child process, Engine.dispose() should be called so that the engine creates brand new database connections local to that fork. Database connections generally do not travel across process boundaries.
That is for SQLAlchemy, but the same should apply to Peewee. Just ensure that all your connections are closed (pool.close_all()) when your workers first start running. Or similarly, if you're opening any database connections at module scope, ensure you call close_all() after using them. This way when your workers start up they will each have an empty pool of connections.
db = FlaskDB(app)
# ...
db.database.close_all()

2006, 'SSL connection error: unknown error number'

I'm using Anaconda and python3.7 and I'm trying to connect to a remote database using the following code:
import MySQLdb
myDB = MySQLdb.connect(host="xxxxx", port=3306, user="xxx",password="xxxx",db="xxxx")
but I get the following error:
File "C:\Users\zanto\anaconda3\lib\site-packages\MySQLdb\connections.py", line 208, in __init__
super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (2006, 'SSL connection error: unknown error number')
I tried 2 users in mysql one using % and one using localhost but I still get the same error.
It was a problem of mySQLdb library.
I installed pymysql library through anaconda and now it's working!
My new code is:
import pymysql.cursors
connection = pymysql.connect(host='xxxx',port=3306,user="xxxx",password="xxxx",db="xxxx", cursorclass=pymysql.cursors.DictCursor)
try:
with connection.cursor() as cursor:
sql="SELECT * FROM Table WHERE Field = 'value'"
cursor.execute(sql)
results = cursor.fetchall()
#print (results) #if you remove the comment you will get the query result as a dictionary
for record in results:
record_line = " ".join('{0}{1}'.format(field,value) for field,value in record.items())
print(record_line)
finally:
connection.close()
More info: A cursor allows Python code to execute MySQL command in a database session. A cursor is created by the connection.cursor() method: they are bound to the connection for the entire lifetime and all the commands are executed in the context of the
database session wrapped by the connection.
The cursor.execute() method runs a query in MySQL database. The cursor.fetchall() method returns the results of a query in list form that contains the records as dictionaries.

Pymysql can't connect do MySQL DB: hangs at "self.sock.recv_into(b)"?

Currently on macOS 10.15 using python 3.7, with MySQL 8.0.19 installed. Developing in VScode with a virtual environment setup. I've created a local database in phpmyadmin as well. I want to connect to it anyway possible. Script:
import pymysql
print("Before")
connection = pymysql.connect(host='localhost',
user='myUserName', password='myPassword', db='db_name', charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
print("After")
When I run the script, execution hangs indefinitely. After printing "After", I have to quit execution. Trackback is:
Traceback (most recent call last):
File "connect.py", line 5, in <module>
user='myUserName', password='myPassword', db='db_name', charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
File "/Users/name/Documents/Work/Connection Test/env/lib/python3.7/site-
packages/pymysql/__init__.py", line 94, in Connect
return Connection(*args, **kwargs)
File "/Users/name/Documents/Work/Connection Test/env/lib/python3.7/site-
packages/pymysql/connections.py", line 325, in __init__
self.connect()
File "/Users/name/Documents/Work/Connection Test/env/lib/python3.7/site-
packages/pymysql/connections.py", line 598, in connect
self._get_server_information()
File "/Users/name/Documents/Work/Connection Test/env/lib/python3.7/site-
packages/pymysql/connections.py", line 975, in _get_server_information
packet = self._read_packet()
File "/Users/name/Documents/Work/Connection Test/env/lib/python3.7/site-
packages/pymysql/connections.py", line 657, in _read_packet
packet_header = self._read_bytes(4)
File "/Users/name/Documents/Work/Connection Test/env/lib/python3.7/site-
packages/pymysql/connections.py", line 691, in _read_bytes
data = self._rfile.read(num_bytes)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/
python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
KeyboardInterrupt
No idea why this is occurring. What is happening here? I have searched a ton of questions on stack overflow and none helped me.
Problem was solved by adding the "unix_socket" param in pymysql connect function. See answer here: pymysql can't connect to MySQL on localhost
import pymysql
print("Before")
conn = pymysql.connect(db='w_dev', user='root', passwd='celebrate',
unix_socket="/tmp/mysql.sock")
print("After")
This script worked.

Query my database on AWS over an SSH tunnel with Pandas

I am trying to query to my database running on AWS over an SSH tunnel with Pandas. But I fail login. Also there is one key that I use to connect on SQL client (Sequel Pro) that I don't know how to use in my python code.
This is the key: heroes-us-west-1-master.cheme3bzdejb.us-west-1.rds.amazonaws.com
This is my code:
with SSHTunnelForwarder(
(host, 22),
ssh_username=ssh_username,
ssh_private_key=ssh_private_key,
remote_bind_address=(localhost, 3306)
) as server:
conn = db.connect(host=localhost,
port=server.local_bind_port,
user=user,
passwd=password,
db=database)
return pd.read_sql_query(q, conn)
df = query("SELECT stores.name, stores.address\
FROM stores;")
This is the error message:
Traceback (most recent call last):
File "connection.py", line 32, in <module>
FROM stores;")
File "connection.py", line 22, in query
remote_bind_address=(localhost, 3306)
File "/Users/Solal/Desktop/SQL/venv/lib/python3.6/site-packages/sshtunnel.py", line 904, in __init__
logger=self.logger
File "/Users/Solal/Desktop/SQL/venv/lib/python3.6/site-packages/sshtunnel.py", line 1095, in _consolidate_auth
raise ValueError('No password or public key available!')
ValueError: No password or public key available!

DAL connection string for Web2Py to MySQL on TurnkeyLinux

On TurnkeyLinux I have tried to connect a Web2Py app to the MySQL instance (which is up and running) with the following DAL statement (in db.py):
db = DAL('mysql://root:pwd2sql#web2py/dbname', fake_migrate_all=True)
It doesn't work and throws the following ticket:
Failure to connect, tried 5 times: Traceback (most recent call last): File "/var/www/web2py/gluon/dal.py", line 7562, in init self.adapter = ADAPTERSself._dbname File "/var/www/web2py/gluon/dal.py", line 2572, in __init_ if do_connect: self.reconnect() File "/var/www/web2py/gluon/dal.py", line 606, in reconnect self.connection = f() File "/var/www/web2py/gluon/dal.py", line 2570, in connector return self.driver.connect(*driver_args) File "/usr/lib/python2.7/dist-packages/MySQLdb/init.py", line 81, in Connect return Connection(args, *kwargs) File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in init super(Connection, self).init(args, **kwargs2) OperationalError: (2003, "Can't connect to MySQL server on 'web2py' (111)")
Does anybody have a connection string sample I can use ?