I have this code for connect to MySQL through a SSH, inside of a python class:
def executeQuery(self, query_string):
print("connecting to database " + self.sql_main_database)
with SSHTunnelForwarder(
(
self.ssh_host,
self.ssh_port),
ssh_username = self.ssh_user,
ssh_pkey = self.pkey,
remote_bind_address=(self.sql_hostname, self.sql_port)
) as tunnel:
print("performing connection")
conn = pymysql.connect(
host="127.0.0.1",
user=self.sql_username,
password=self.sql_password,
db=self.sql_main_database,
port=tunnel.local_bind_port)
query = query_string
print("Querying")
data = pd.read_sql_query(query, conn)
print("Done!")
conn.close()
return data
The code is working well, but when the query is not well defined, the notebook freezes.
Then, I tried to use a try/catch, and the code ended like this
def executeQuery(self, query_string):
try:
with SSHTunnelForwarder(
(
self.ssh_host,
self.ssh_port
),
ssh_username = self.ssh_user,
ssh_pkey = self.pkey,
remote_bind_address=(self.sql_hostname, self.sql_port)
) as tunnel:
try:
conn = pymysql.connect(
host = "127.0.0.1",
user = self.sql_username,
password = self.sql_password,
db = self.sql_main_database,
port = tunnel.local_bind_port
)
try:
query = query_string
data = pd.read_sql_query(query, conn)
return data
except DatabaseError as e:
Log.debug(self,str(e))
raise DatabaseError
except pymysql.err.InternalError as e:
Log.debug(self, str(e))
raise DataError
except Exception as e:
Log.debug(self, "[Error]Setting up database: \'" + self.sql_main_database + "\'")
raise DataError
The issue is that pd.read_sql_query never stops so the except is never called, the try won't fail, and the process will just continue forever
The timeout workaround is not possible, because the queries don't have defined execution times and some of them can stay in processing during a couple of hours.
I'm not sure how to fix it.
Indeed the problem was not on the connector, just updating the jupyter version was needed.
Related
import serial
try:
ser = serial.Serial(port = 'COM7', baudrate = 921600, timeout = 2, bytesize = 8)
except serial.SerialException:
ser.close()
ser = serial.Serial(port = 'COM7', baudrate = 921600, timeout = 2, bytesize = 8)
Sometimes my script exits before reaching ser.close() at the end so I thought the below code would catch the exception, close the port and reopen the port:
However, I got "NameError: name 'ser' is not defined" at ser.close() statement when running the whole script.
But strange enough, there is no issues if I highlighted the code section and run as a section.
My Shiny App was crushing when wrong connection credentials were passed to the connection string. I then put my connection string within a tryCatch as follows:
,,,
ConnectToDb <- function(){
con <- tryCatch({
dbConnect(MySQL(),
user = input$db_user,
password = input$db_password,
dbname = input$db_name,
host = input$db_host,
port = input$db_port)
print("Connection made")
####
sql <- "SELECT * FROM PMSAnalytics;"
data <- dbGetQuery(con, sql)
# # Disconnect from the DB
dbDisconnect(con)
# # Convert to data.frame
data <- data.frame(data)
data$timestamp <- as_datetime(now())
data
####
}, error = function(e) {
message('Please confirm your login details')
print(e)
},
warning = function(w){
message('A warning has occured')
print(w)
return(NA)
}
)
}
,,,
Now the application does not crush, but however no error messages or warning are passed when wrong credentials are used and neither do I get a connection success. I have checked this site for similar questions, but I seem not to get any. Kindly assist with polishing the code.
Regards,
Chris
I work with showNotification, which directly shows a notification in the shiny UI, you could also use it for the connection success.
Also, Options for this are,
duration = 60 (in this case for 60 seconds)
closeButton = FALSE
For example:
error = function(e) {
showNotification(paste0(e), type = 'error')
}
warning = function(w){
showNotification(paste0(w), type = 'warning')
return(NA)
}
Python:3.8.5
sshtunnel:0.2.1
mysqlclient:1.4.6
mysql-connector:2.2.9
I am using SSHTunnelForwarder to retrieve data from a Mysql database.
Here is the script I use to connect via SSH to the DB:
elif self._remote == 1:
with SSHTunnelForwarder(
(self._host, 22),
ssh_password = self._ssh_password,
ssh_username = self._ssh_login,
remote_bind_address = (self._remote_bind_address, 3306)) as server:
print('Connection:',server.local_bind_address)
cnx = MySQLdb.connect(host = '127.0.0.1',
port = server.local_bind_port,
user = self._db_user,
passwd = self._db_password,
db = self._db_name)
cursor = cnx.cursor()
res = pd.read_sql(request, con = cnx)
cursor.close()
cnx.close()
An example request could be in the following form:
request = 'SELECT * FROM conjunctions AS c LEFT JOIN events AS e ON e.eventId=c.eventId ORDER BY e.eventId;'
The script returns me a valid response, but will not exit to shell.
a threading.enumerate() will print this:
[<_MainThread(MainThread, started 139701046208320)>, <paramiko.Transport at 0x74850ac0 (unconnected)>, <paramiko.Transport at 0xae9e4e80 (unconnected)>]
I have found this issue relating to the same problem, however suggested solutions are not working for me.
Manually closing the tunnel with a server.stop() does not work.
Adding ssh_server.daemon_forward_servers = True as suggested in the issue mentioned above does not work.
Most of all, this problem appears approx 4/5 times the script is launched.
Any help to understand what is going on would be greatly appreciated.
Thank you.
I am trying to run the following code to populate a table in parallel for a certain application. First the following function is defined which is supposed to connect to my db and execute the sql command with the values given (to insert into table).
def dbWriter(sql, rows) :
# load cnf file
MYSQL_CNF = os.path.abspath('.') + '/mysql.cnf'
conn = MySQLdb.connect(db='dedupe',
charset='utf8',
read_default_file = MYSQL_CNF)
cursor = conn.cursor()
cursor.executemany(sql, rows)
conn.commit()
cursor.close()
conn.close()
And then there is this piece:
pool = dedupe.backport.Pool(processes=2)
done = False
while not done :
chunks = (list(itertools.islice(b_data, step)) for step in
[step_size]*100)
results = []
for chunk in chunks :
print len(chunk)
results.append(pool.apply_async(dbWriter,
("INSERT INTO blocking_map VALUES (%s, %s)",
chunk)))
for r in results :
r.wait()
if len(chunk) < step_size :
done = True
pool.close()
Everything works and there are no errors. But at the end, my table is empty, meaning somehow the insertions were not successful. I have tried so many things to fix this (including adding column names for insertion) after many google searches and have not been successful. Any suggestions would be appreciated. (running code in python2.7, gcloud (ubuntu). note that indents may be a bit messed up after pasting here)
Please also note that "chunk" follows exactly the required data format.
Note. This is part of this example
Please note that the only thing I am changing in the above example (linked) is that I am separating the steps for creation of and inserting into the tables since I am running my code on gcloud platform and it enforces GTID standards.
Solution was changing dbwriter function to:
conn = MySQLdb.connect(host = # host ip,
user = # username,
passwd = # password,
db = 'dedupe')
cursor = conn.cursor()
cursor.executemany(sql, rows)
cursor.close()
conn.commit()
conn.close()
I have a python script which everytime a wait_for_page call is made it writes the time it took to wait for the page to a database. The query is below:
conn = mysql.connector.connect(**config)
connect = conn.cursor()
params = {'build': self.tc.tag, 'page': unicode(self), 'object_id': self.object_id, 'page_header':
self.page_header, 'interval': t.interval, 'timestamp': timestamp}
query = u'INSERT INTO page_load_times (build, page, object_id, page_header, elapsed_time, date_run) ' \
'VALUES (%(build)s, %(page)s, %(object_id)s, %(page_header)s, %(interval)s, %(timestamp)s)'
connect.execute(query, params)
conn.commit()
conn.close()
Occasionally, when this runs, I get an error which says:
"Failed processing pyformat-parameters; %s" % err)
ProgrammingError: Failed processing pyformat-parameters; 'MySQLConverter'
object has no attribute '_list_to_mysql'
I know what is causing this, just uncertain how to go about fixing it. The 'page': unicode(self) param occasionally gets a list as a result.
In an attempt to fix this, I tweaked the above script to extract the list into a string, with the following:
page_list = u'{}'.format(self)
page_results = "('%s')" % "','".join(page_list)
params = {'build': self.tc.tag, 'page': page_results, 'object_id': self.object_id, 'page_header':
self.page_header, 'interval': t.interval, 'timestamp': timestamp}
When I run this, the error I am getting now is that the data is too long for the field. I debug it, to find that my page results has each character parsed out individually looking like so:
u'(\\'A\\',\\'p\\',\\'p\\',\\'M\\',\\'a\\',\\'i\\',\\'n\\',\\'M\\',\\'e\\',\\'n\\',\\'u\\',\\':\\',\\' \\',\\'N\\',\\'o\\',\\'n\\',\\'e\\')'
So the solution was to do the following, which takes the page_header and if it is in the instance of list to make that list a string:
conn = mysql.connector.connect(**config)
connect = conn.cursor()
page_list = u'{}'.format(self)
page_header_list = u'{}'.format(self.page_header)
if isinstance(page_header_list, list):
page_header_list = ', '.join(page_header_list)[0:100]
params = {'build': self.tc.tag, 'page': page_list, 'object_id': self.object_id,
'page_header': page_header_list, 'interval': t.interval, 'timestamp': timestamp}
query = u'INSERT INTO page_load_times (build, page, object_id, page_header, elapsed_time, date_run) ' \
'VALUES (%(build)s, %(page)s, %(object_id)s, %(page_header)s, %(interval)s, %(timestamp)s)'
connect.execute(query, params)
conn.commit()
conn.close()
Thank you #DarthOpto, you gave me lots of light. I solved it by putting srt around the variable:
params = {'build': self.tc.tag, 'page': srt(page_list),'object_id': self.object_id,
'page_header': srt(page_header_list),'interval': t.interval,'timestamp': timestamp}