After making a connection with mysql library, i'd like to dowload all the database from the connection in my local space (tranforming them into pandas dataframe).
Here's my code:
import MySQLdb
import pandas as pd
conn = MySQLdb.connect(host='host' , user='datbase', passwd='password', db='databases' )
cursor = conn.cursor()
query = cursor.execute(' SELECT * FROM pro ')
df = pd.read_sql(query , conn)
row = cursor.fetchone()
conn.close()
I finnaly got the connection, so i can make some query. But i'd like to use these sql database as a pandas dataframe, '''how can i do it'''?
Just use
query = ' SELECT * FROM pro '
df = pd.read_sql(query , conn)
And df should already be your desired dataframe.
Related
I am getting this error: DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': Not all parameters were used in the SQL statement. when trying to convert my dataframe to sql
My connection variable:
con = mysql.connector.connect(
host="****",
port="****",
database="*****",
user="*****",
password="*****"
)
My try to convert it to sql:
df.to_sql('menageiro2',con)
Note: I am using:
import pandas as pd
import sqlalchemy
import mysql.connector
The reference says con: sqlalchemy.engine.(Engine or Connection) or sqlite3.Connection. You appear to be passing in a mysql connection instead of a SQLAlchamy engine (that you connected to MySQL):
con = sqlalchemy.create_engine(
'mysql+mysqlconnector://<user>:<password>#<host>:<port>/<default_db>...')
i want to retrieve image from database using python but i have a problem where i execute this code
import mysql.connector
import io
from PIL import Image
connection= mysql.connector.connect(
host ="localhost",
user ="root",
passwd ="pfe_altran",
database = "testes",
)
cursor=connection.cursor()
sql1 = "SELECT * FROM pfe WHERE id = 1 "
cursor.execute(sql1)
data2 = cursor.fetchall()
file_like2 = io.BytesIO(data2[0][0])
img1=Image.open(file_like2)
img1.show()
cursor.close()
connection.close()
and i have this error :
file_like2 = io.BytesIO(data2[0][0]) TypeError: a bytes-like object
is required, not 'int'
cursor.fetchall() returns a list of rows to your variable so you need to handle that using a looper
for row in data2:
file_like2 = io.BytesIO(row[0]) #assumming row[0] contains the byte form of your image
you can use cursor.fetchmany(size=1) or cursor.fetchone() if you know that your query will return only a single row or if you know that you only need one row, this way you can manipulate it directly and not use the loop.
#!/usr/bin/python
print('Content-type: text/html\r\n\r')
import mysql.connector
import json
conn = mysql.connector.connect(
user='root',
password='',
host='localhost',
database='id1914180_myproducts')
cur = conn.cursor()
query = ("SELECT * FROM Reviews")
cur.execute(query)
result=cur.fetchall()
print(json.dumps(result))
cur.close()
conn.close()
I want the data to be displayed in dictionary format like given:
{'UserName':'Rozi','You Me and Dupree':2.5,'Lady in the water':4.0}
I'm using python to export a large matrixs (shape around 3000 * 3000) into MySQL.
Right now I'm using MySQLdb to insert those values but it's too troublesome and too inefficient. Here is my code:
# -*- coding:utf-8 -*-
import MySQLdb
import numpy as np
import pandas as pd
import time
def feature_to_sql_format(df):
df = df.fillna(value='')
columns = list(df.columns)
index = list(df.index)
index_sort = np.reshape([[int(i)] * len(columns) for i in index], (-1)).tolist()
columns_sort = (columns * len(index))
values_sort = df.values.reshape(-1).tolist()
return str(zip(index_sort, columns_sort, values_sort))[1: -1].replace("'NULL'", 'NULL')
if __name__ == '__main__':
t1 = time.clock()
df = pd.read_csv('C:\\test.csv', header=0, index_col=0)
output_string = feature_to_sql_format(df)
sql_CreateTable = 'USE derivative_pool;DROP TABLE IF exists test1;' \
'CREATE TABLE test1(date INT NOT NULL, code VARCHAR(12) NOT NULL, value FLOAT NULL);'
sql_Insert = 'INSERT INTO test (date,code,value) VALUES ' + output_string + ';'
con = MySQLdb.connect(......)
cur = con.cursor()
cur.execute(sql_CreateTable)
cur.close()
cur = con.cursor()
cur.execute(sql_Insert)
cur.close()
con.commit()
con.close()
t2 = time.clock()
print t2 - t1
And it consumes around 274 seconds totally.
I was wondering if there is a simplier way to do this, I thought of export the matrix to csv and then use LOAD DATA INFILE to import, but it's also too complicated.
I noticed that in pandas documentation pandas dataframe has a function to_sql, and in version 0.14 you can set the 'flavor' to 'mysql', that is:
df.to_sql(con=con, name=name, flavor='mysql')
But now my pandas version is 0.19.2 and the flavor is reduced to only 'sqlite'...... And I still tried to use
df.to_sql(con=con, name=name, flavor='sqlite')
and it gives me an error.
Is there any convinient way to do this?
Later pandas versions support SQLalchemy connectors instead of flavor = "mysql"
First, install dependencies:
pip install mysql-connector-python-rf==2.2.2
pip install MySQL-python==1.2.5
pip install SQLAlchemy==1.1.1
Then create the engine:
from sqlalchemy import create_engine
connection_string= "mysql+mysqlconnector://root:#localhost/MyDatabase"
engine = create_engine(connection_string)
Then you can use df.to_sql(...):
df.to_sql('MyTable', engine)
Here are some things you can do in MYSQL to speed up your data load:
SET FOREIGN_KEY_CHECKS = 0;
SET UNIQUE_CHECKS = 0;
SET SESSION tx_isolation='READ-UNCOMMITTED';
SET sql_log_bin = 0;
#LOAD DATA LOCAL INFILE....
SET UNIQUE_CHECKS = 1;
SET FOREIGN_KEY_CHECKS = 1;
SET SESSION tx_isolation='READ-REPEATABLE';
How can I escape the input to a MySQL db in Python3?
I'm using PyMySQL and works fine, but when I try to do something like:
cursor.execute("SELECT * FROM `Codes` WHERE `ShortCode` = '{}'".format(request[1]))
it won't work if the string has ' or ". I also tried:
cursor.execute("SELECT * FROM `Codes` WHERE `ShortCode` = %s",request[1])
The problem with this is that the library (PyMySQL) uses the formatting syntax for Python2.x, %, that doesn't work anymore.
I also found this possible solution
conn.escape_string()
in here, but I don't know where to add this code.
This is all I got:
import pymysql
import sys
conn = pymysql.connect( host = "localhost",
user = "test",
passwd = "",
db = "test")
cursor = conn.cursor()
cursor.execute("SELECT * FROM `Codes` WHERE `ShortCode` = {}".format(request[1]))
result = cursor.fetchall()
cursor.close()
conn.close()
Edit: I solved it! In PyMySQL the right way is like this:
import pymysql
import sys
conn = pymysql.connect(host="localhost",
user="test",
passwd="",
db="test")
cursor = conn.cursor()
text = conn.escape(request[1])
cursor.execute("SELECT * FROM `Codes` WHERE `ShortCode` = {}".format(text))
cursor.close()
conn.close()
Where the text = conn.escape(request[1]) line is what escapes the code. Found it inside PyMySQL code. There, request[1] is the input.
Although the "solved" answer works, it is not best practice. When using a library conforming to the Python DBI, you should be using bind variables rather than formatting a string and passing it to execute. There are dangers inherent in that methodology.
Therefore, this is the right way to do it:
cursor.execute("SELECT * FROM `Codes` WHERE `ShortCode` = %s", text)
Note that this is not a format string but a bind variable passed to the executing cursor.
For details: Python DBI PEP
Solved. In PyMySQL the right way is like this:
import pymysql
import sys
conn = pymysql.connect(host="localhost",
user="test",
passwd="",
db="test")
cursor = conn.cursor()
text = conn.escape(request[1])
cursor.execute("SELECT * FROM `Codes` WHERE `ShortCode` = {}".format(text))
cursor.close()
conn.close()
Where the text = conn.escape(request[1]) line is what escapes the code. Found it inside PyMySQL code. There, request[1] is the input.
Ready to use helper function
def mysql_insert(conn, table, row):
cols = ', '.join('`{}`'.format(col) for col in row.keys())
vals = ', '.join('%({})s'.format(col) for col in row.keys())
sql = 'INSERT INTO `{0}` ({1}) VALUES ({2})'.format(table, cols, vals)
conn.cursor().execute(sql, row)
conn.commit()
Usage example
insert_into(conn, 'people', {
'firstname': 'John',
'lastname': 'Doe',
'age': 18, })
Reference: https://github.com/PyMySQL/PyMySQL/blob/master/pymysql/cursors.py#L157-L158
def execute(self, query, args=None):
If args is a list or tuple, %s can be used as a placeholder in the query.
If args is a dict, %(name)s can be used as a placeholder in the query.