Python:3.8.5
sshtunnel:0.2.1
mysqlclient:1.4.6
mysql-connector:2.2.9
I am using SSHTunnelForwarder to retrieve data from a Mysql database.
Here is the script I use to connect via SSH to the DB:
elif self._remote == 1:
with SSHTunnelForwarder(
(self._host, 22),
ssh_password = self._ssh_password,
ssh_username = self._ssh_login,
remote_bind_address = (self._remote_bind_address, 3306)) as server:
print('Connection:',server.local_bind_address)
cnx = MySQLdb.connect(host = '127.0.0.1',
port = server.local_bind_port,
user = self._db_user,
passwd = self._db_password,
db = self._db_name)
cursor = cnx.cursor()
res = pd.read_sql(request, con = cnx)
cursor.close()
cnx.close()
An example request could be in the following form:
request = 'SELECT * FROM conjunctions AS c LEFT JOIN events AS e ON e.eventId=c.eventId ORDER BY e.eventId;'
The script returns me a valid response, but will not exit to shell.
a threading.enumerate() will print this:
[<_MainThread(MainThread, started 139701046208320)>, <paramiko.Transport at 0x74850ac0 (unconnected)>, <paramiko.Transport at 0xae9e4e80 (unconnected)>]
I have found this issue relating to the same problem, however suggested solutions are not working for me.
Manually closing the tunnel with a server.stop() does not work.
Adding ssh_server.daemon_forward_servers = True as suggested in the issue mentioned above does not work.
Most of all, this problem appears approx 4/5 times the script is launched.
Any help to understand what is going on would be greatly appreciated.
Thank you.
Related
My Shiny App was crushing when wrong connection credentials were passed to the connection string. I then put my connection string within a tryCatch as follows:
,,,
ConnectToDb <- function(){
con <- tryCatch({
dbConnect(MySQL(),
user = input$db_user,
password = input$db_password,
dbname = input$db_name,
host = input$db_host,
port = input$db_port)
print("Connection made")
####
sql <- "SELECT * FROM PMSAnalytics;"
data <- dbGetQuery(con, sql)
# # Disconnect from the DB
dbDisconnect(con)
# # Convert to data.frame
data <- data.frame(data)
data$timestamp <- as_datetime(now())
data
####
}, error = function(e) {
message('Please confirm your login details')
print(e)
},
warning = function(w){
message('A warning has occured')
print(w)
return(NA)
}
)
}
,,,
Now the application does not crush, but however no error messages or warning are passed when wrong credentials are used and neither do I get a connection success. I have checked this site for similar questions, but I seem not to get any. Kindly assist with polishing the code.
Regards,
Chris
I work with showNotification, which directly shows a notification in the shiny UI, you could also use it for the connection success.
Also, Options for this are,
duration = 60 (in this case for 60 seconds)
closeButton = FALSE
For example:
error = function(e) {
showNotification(paste0(e), type = 'error')
}
warning = function(w){
showNotification(paste0(w), type = 'warning')
return(NA)
}
I have this code for connect to MySQL through a SSH, inside of a python class:
def executeQuery(self, query_string):
print("connecting to database " + self.sql_main_database)
with SSHTunnelForwarder(
(
self.ssh_host,
self.ssh_port),
ssh_username = self.ssh_user,
ssh_pkey = self.pkey,
remote_bind_address=(self.sql_hostname, self.sql_port)
) as tunnel:
print("performing connection")
conn = pymysql.connect(
host="127.0.0.1",
user=self.sql_username,
password=self.sql_password,
db=self.sql_main_database,
port=tunnel.local_bind_port)
query = query_string
print("Querying")
data = pd.read_sql_query(query, conn)
print("Done!")
conn.close()
return data
The code is working well, but when the query is not well defined, the notebook freezes.
Then, I tried to use a try/catch, and the code ended like this
def executeQuery(self, query_string):
try:
with SSHTunnelForwarder(
(
self.ssh_host,
self.ssh_port
),
ssh_username = self.ssh_user,
ssh_pkey = self.pkey,
remote_bind_address=(self.sql_hostname, self.sql_port)
) as tunnel:
try:
conn = pymysql.connect(
host = "127.0.0.1",
user = self.sql_username,
password = self.sql_password,
db = self.sql_main_database,
port = tunnel.local_bind_port
)
try:
query = query_string
data = pd.read_sql_query(query, conn)
return data
except DatabaseError as e:
Log.debug(self,str(e))
raise DatabaseError
except pymysql.err.InternalError as e:
Log.debug(self, str(e))
raise DataError
except Exception as e:
Log.debug(self, "[Error]Setting up database: \'" + self.sql_main_database + "\'")
raise DataError
The issue is that pd.read_sql_query never stops so the except is never called, the try won't fail, and the process will just continue forever
The timeout workaround is not possible, because the queries don't have defined execution times and some of them can stay in processing during a couple of hours.
I'm not sure how to fix it.
Indeed the problem was not on the connector, just updating the jupyter version was needed.
I am trying to run the following code to populate a table in parallel for a certain application. First the following function is defined which is supposed to connect to my db and execute the sql command with the values given (to insert into table).
def dbWriter(sql, rows) :
# load cnf file
MYSQL_CNF = os.path.abspath('.') + '/mysql.cnf'
conn = MySQLdb.connect(db='dedupe',
charset='utf8',
read_default_file = MYSQL_CNF)
cursor = conn.cursor()
cursor.executemany(sql, rows)
conn.commit()
cursor.close()
conn.close()
And then there is this piece:
pool = dedupe.backport.Pool(processes=2)
done = False
while not done :
chunks = (list(itertools.islice(b_data, step)) for step in
[step_size]*100)
results = []
for chunk in chunks :
print len(chunk)
results.append(pool.apply_async(dbWriter,
("INSERT INTO blocking_map VALUES (%s, %s)",
chunk)))
for r in results :
r.wait()
if len(chunk) < step_size :
done = True
pool.close()
Everything works and there are no errors. But at the end, my table is empty, meaning somehow the insertions were not successful. I have tried so many things to fix this (including adding column names for insertion) after many google searches and have not been successful. Any suggestions would be appreciated. (running code in python2.7, gcloud (ubuntu). note that indents may be a bit messed up after pasting here)
Please also note that "chunk" follows exactly the required data format.
Note. This is part of this example
Please note that the only thing I am changing in the above example (linked) is that I am separating the steps for creation of and inserting into the tables since I am running my code on gcloud platform and it enforces GTID standards.
Solution was changing dbwriter function to:
conn = MySQLdb.connect(host = # host ip,
user = # username,
passwd = # password,
db = 'dedupe')
cursor = conn.cursor()
cursor.executemany(sql, rows)
cursor.close()
conn.commit()
conn.close()
i am new to python and mysql,
i have to present a code to my office head. this code should help pull a table called "customer". I just want assistance to know if there are any errors in my code.
hostname = 'localhost'
port =
username = 'USERNAME'
password = 'PASSWORD'
database = 'DBNAME'
# Open database connection
conn = pymysql.connect(host=hostname, port=,user=username, passwd=password, db=database)
cur = conn.cursor()
cur.execute("SELECT * customer")
print(cur)
db.close()
Thanks in Advance,
Ninad.
I am using python 3.6
You may need to print
cur.fetchall()
to have your results appear
I was wondering if someone could help with this annoying issue.
I'm trying to create/make multiple connections to different database.
I have a data.frame with 3 connection credentials named conf - It works if I manually enter the connections variable like so:
conn <- dbConnect(MySQL(), user=conf$user, password=conf$passws, host=conf$host, dbname=conf$db)
which ends up creating a single connection.
However, what I want is to be able to refer to the connection as:
conf$conn <- dbConnect(MySQL(), user=conf$user, password=conf$passws, host=conf$host, dbname=conf$db)
here is the error message I'm getting.
Error in rep(value, length.out = nrows) :
attempt to replicate an object of type 'S4'
I think the problem is how I'm adding conf$conn
I used a combination of the pool and config package to solve a similar problem to set up a number of simultaneous PostgreSQL connections. Note that this solution needs a config.yml file with the connection properties for db1 and db2.
library(pool)
library(RPostgreSQL)
connect <- function(cfg) {
config <- config::get(config = cfg)
dbPool(
drv = dbDriver("PostgreSQL", max.con = 100),
dbname = config$dbname,
host = config$host,
port = config$port,
user = config$user,
password = config$password
)
}
conn <- lapply(c("db1", "db2"), connect)