How can I escape the input to a MySQL db in Python3? - mysql

How can I escape the input to a MySQL db in Python3?
I'm using PyMySQL and works fine, but when I try to do something like:
cursor.execute("SELECT * FROM `Codes` WHERE `ShortCode` = '{}'".format(request[1]))
it won't work if the string has ' or ". I also tried:
cursor.execute("SELECT * FROM `Codes` WHERE `ShortCode` = %s",request[1])
The problem with this is that the library (PyMySQL) uses the formatting syntax for Python2.x, %, that doesn't work anymore.
I also found this possible solution
conn.escape_string()
in here, but I don't know where to add this code.
This is all I got:
import pymysql
import sys
conn = pymysql.connect( host = "localhost",
user = "test",
passwd = "",
db = "test")
cursor = conn.cursor()
cursor.execute("SELECT * FROM `Codes` WHERE `ShortCode` = {}".format(request[1]))
result = cursor.fetchall()
cursor.close()
conn.close()
Edit: I solved it! In PyMySQL the right way is like this:
import pymysql
import sys
conn = pymysql.connect(host="localhost",
user="test",
passwd="",
db="test")
cursor = conn.cursor()
text = conn.escape(request[1])
cursor.execute("SELECT * FROM `Codes` WHERE `ShortCode` = {}".format(text))
cursor.close()
conn.close()
Where the text = conn.escape(request[1]) line is what escapes the code. Found it inside PyMySQL code. There, request[1] is the input.

Although the "solved" answer works, it is not best practice. When using a library conforming to the Python DBI, you should be using bind variables rather than formatting a string and passing it to execute. There are dangers inherent in that methodology.
Therefore, this is the right way to do it:
cursor.execute("SELECT * FROM `Codes` WHERE `ShortCode` = %s", text)
Note that this is not a format string but a bind variable passed to the executing cursor.
For details: Python DBI PEP

Solved. In PyMySQL the right way is like this:
import pymysql
import sys
conn = pymysql.connect(host="localhost",
user="test",
passwd="",
db="test")
cursor = conn.cursor()
text = conn.escape(request[1])
cursor.execute("SELECT * FROM `Codes` WHERE `ShortCode` = {}".format(text))
cursor.close()
conn.close()
Where the text = conn.escape(request[1]) line is what escapes the code. Found it inside PyMySQL code. There, request[1] is the input.

Ready to use helper function
def mysql_insert(conn, table, row):
cols = ', '.join('`{}`'.format(col) for col in row.keys())
vals = ', '.join('%({})s'.format(col) for col in row.keys())
sql = 'INSERT INTO `{0}` ({1}) VALUES ({2})'.format(table, cols, vals)
conn.cursor().execute(sql, row)
conn.commit()
Usage example
insert_into(conn, 'people', {
'firstname': 'John',
'lastname': 'Doe',
'age': 18, })
Reference: https://github.com/PyMySQL/PyMySQL/blob/master/pymysql/cursors.py#L157-L158
def execute(self, query, args=None):
If args is a list or tuple, %s can be used as a placeholder in the query.
If args is a dict, %(name)s can be used as a placeholder in the query.

Related

Draw a graph from a SQL request using Dash

I have been searching but I didn't find a simple way to draw a graph from a SQL request.
For example I have this code, and I want to make a bar chart from the result of the request :
import pymysql as sql
from dash import dcc
DB = ...
HOST = ...
USER = ...
PASSWORD = ...
connection = sql.connect(host=HOST,
port=x,
user=USER,
password=PASSWORD,
database=DB,
cursorclass=sql.cursors.DictCursor)
with connection.cursor() as cursor:
# Read a single record
sql = 'SELECT COUNT(*) AS count FROM table'
cursor.execute(sql)
result = cursor.fetchone()
print(result)
Also, I would like to update the chart regularly.
Thank you

CSV read into MySQLdb failing

I am having a problem with reading my csv file into the MySQL database. I have tried a number of solutions, but the errors just keep changing and the code isn't working. This same code had worked with another csv file, so I'm thinking I might be doing something wrong with this one?
Here is my code
from database_access import *
from builtins import bytes, int, str
import codecs
import csv
import requests
from urllib.parse import urlparse, urljoin
from bs4 import BeautifulSoup
import re
import cgi
import MySQLdb
import chardet
# from database_access import *
import MySQLdb
import simplejson
if __name__ == '__main__':
with open("SIMRA.csv",'r') as file:
reader = csv.reader(file)
#reader = csv.reader(text)
next(reader, None)
print ("project running")
#print (row[7])
#rowlist = []
all_links = []
all_project_ids = []
for row in reader:
if row[7] != "" and row[16] != "":
country = row[2]
city = row[8]
description = row[11] + '' + row[12]
title = row[7].replace("'", "''")
link = row[16]
#date_start = row[9]
#print a check here
print(title,description,country, city, link)
db = MySQLdb.connect(host, username, password, database, charset='utf8')
cursor = db.cursor()
new_project = True
proj_check = "SELECT * from Projects where ProjectName like '%" + title + "%'"
#proj_check = "SELECT * from Projects where ProjectName like %s",(title,)
#cur.execute("SELECT * FROM records WHERE email LIKE %s", (search,))
cursor.execute(proj_check)
num_rows = cursor.rowcount
if num_rows != 0:
new_project = False
url_compare = "SELECT * from Projects where ProjectWebpage like '" + link + "'"
#url_compare = "SELECT * from Projects where ProjectWebpage like %s",(link,)
cursor.execute(url_compare)
num_rows = cursor.rowcount
if num_rows != 0:
new_project = False
if new_project:
project_insert = "Insert into Projects (ProjectName,ProjectWebpage,FirstDataSource,DataSources_idDataSources) VALUES (%s,%s,%s,%s)"
cursor.execute(project_insert, (title, link,'SIMRA', 5))
projectid = cursor.lastrowid
print(projectid)
#ashoka_projectids.append(projectid)
db.commit()
ins_desc = "Insert into AdditionalProjectData (FieldName,Value,Projects_idProjects,DateObtained) VALUES (%s,%s,%s,NOW())"
cursor.executemany(ins_desc, ("Description", description, str(projectid)))
db.commit()
ins_location = "Insert into ProjectLocation (Type,Country,City,Projects_idProjects) VALUES (%s,%s,%s,%s)"
cursor.execute(ins_location, ("Main", country,city, str(projectid)))
db.commit()
else:
print('Project already exists!')
print(title)
all_links.append(link)
#print out SIMRA's links to a file for crawling later
with open('simra_links', 'w', newline='') as f:
write = csv.writer(f)
for row in all_links:
columns = [c.strip() for c in row.strip(', ').split(',')]
write.writerow(columns)
When I ran this, I got the following error:
File "/usr/lib/python3.8/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 898: invalid start byte
I did some research and tried handling the encoding error by adding different forms of encoding, as seen here - UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0xa5 in position 0: invalid start byte, and Python MySQLdb TypeError: not all arguments converted during string formatting. Added this in this in the csv open parameter -
with open("SIMRA.csv", 'r', encoding="cp437", errors='ignore') as file:
Running the code with these different encoding options came up with a different error:
MySQLdb._exceptions.ProgrammingError: not all arguments converted during bytes formatting
Further research suggested using tuples or lists in order to address this problem, so I added these in the 'select' function in the code, as suggested here - Python MySQLdb TypeError: not all arguments converted during string formatting and in the Python SQL documentation here - PythonMySqldb
So the select query became:
proj_check = "SELECT * from Projects where ProjectName like %s",(title,)
cursor.execute(proj_check)
num_rows = cursor.rowcount
if num_rows != 0:
new_project = False
url_compare = "SELECT * from Projects where ProjectWebpage like %s",(link,)
cursor.execute(url_compare)
num_rows = cursor.rowcount
if num_rows != 0:
new_project = False
When I ran the code, I came up with this Assertion Error and I have no idea what to do anymore.
File "/home/ros/.local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 205, in execute
assert isinstance(query, (bytes, bytearray))
AssertionError
I have run out of ideas. It might be that I'm missing something small, but I can't figure this out now as I've been battling with this for two days now.
Can anyone help point out what I'm missing? It will be greatly appreciated. This code ran perfectly with another csv file. I am running this with Python 3.8 btw.
Have solved this now. I had to use a different encoding with the original code and this solved the problem. So, I changed the csv open parameter to:
with open("SIMRA.csv",'r', encoding="ISO-8859-1") as file:
reader = csv.reader(file)
Were you expecting £? You need to specify what the encoding of the file is. It may be "latin1". See the syntax of LOAD DATA for how to specify CHARACTER SET latin1.

Python constructor confusion

I'm playing with python trying to create a basic repository class (normally a C++/C# for work) and am having an issue.
The following code has bombs out on
a = officesRepo(conn) saying "Too many positional arguments for constructor call", but it's being given the only argument specified, the MySql connection object.
I'm coding in vscode on linux using python3.8. I'm wondering if pylint is expecting me to pass in "self", when I don't think it's needed.
Any help/advice/tips greatly received. Flame away if you like, as long as it teaches me something! ;-)
import pymysql.cursors
import Pocos
class officesRepo:
def __init__(conn):
self.conn = conn
def create(office):
pass
def getAll():
cursor = conn.cursor()
SQL = "SELECT `officeCode`, `city`, `phone`, `addressLine1`, `addressLine2`, `state`, `country`, `postalCode`, `territory` "
SQL += "FROM `offices`"
cursor.execute(SQL)
#result = cursor.fetchone()
ret = []
for val in cursor:
ret.append(ret.append(val["officeCode"], val["city"], val["phone"], val["addressLine1"], val["addressLine2"], val["state"], val["country"], val["postalCode"], val["territory"]))
return ret
def getById(id):
pass
conn = pymysql.connect(host='localhost',
user='user',
password='password',
db='classicmodel',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
a = officesRepo(conn)
b = a.getAll()
print(b)
The first parameter of an instance method is self. You don't need to pass it explicitly, but you do need to include it in the parameter list. Right now though, the conn parameter is acting as self, then there's no other parameters after that (thus the error).
You'd need
def __init__(self, conn):
. . .
then similarly for the other methods. All instance methods require an explicit self parameter.

How to download data from mysql connection

After making a connection with mysql library, i'd like to dowload all the database from the connection in my local space (tranforming them into pandas dataframe).
Here's my code:
import MySQLdb
import pandas as pd
conn = MySQLdb.connect(host='host' , user='datbase', passwd='password', db='databases' )
cursor = conn.cursor()
query = cursor.execute(' SELECT * FROM pro ')
df = pd.read_sql(query , conn)
row = cursor.fetchone()
conn.close()
I finnaly got the connection, so i can make some query. But i'd like to use these sql database as a pandas dataframe, '''how can i do it'''?
Just use
query = ' SELECT * FROM pro '
df = pd.read_sql(query , conn)
And df should already be your desired dataframe.

How to retrieve image from database using python

i want to retrieve image from database using python but i have a problem where i execute this code
import mysql.connector
import io
from PIL import Image
connection= mysql.connector.connect(
host ="localhost",
user ="root",
passwd ="pfe_altran",
database = "testes",
)
cursor=connection.cursor()
sql1 = "SELECT * FROM pfe WHERE id = 1 "
cursor.execute(sql1)
data2 = cursor.fetchall()
file_like2 = io.BytesIO(data2[0][0])
img1=Image.open(file_like2)
img1.show()
cursor.close()
connection.close()
and i have this error :
file_like2 = io.BytesIO(data2[0][0]) TypeError: a bytes-like object
is required, not 'int'
cursor.fetchall() returns a list of rows to your variable so you need to handle that using a looper
for row in data2:
file_like2 = io.BytesIO(row[0]) #assumming row[0] contains the byte form of your image
you can use cursor.fetchmany(size=1) or cursor.fetchone() if you know that your query will return only a single row or if you know that you only need one row, this way you can manipulate it directly and not use the loop.