I am trying to connect to a mysql database,
works fine with Option 1:
from sqlalchemy import create_engine
engine = create_engine('mysql://root:root#localhost/lend', echo=True)
cnx = engine.connect()
x = cnx.execute("SELECT * FROM user")
but breaks down here:
from pandas.io import sql
xx = sql.read_frame("SELECT * FROM user", cnx)
cnx.close()
with
AttributeError: 'Connection' object has no attribute 'rollback'
You need to have a raw database connection, and not an instance of Connection. In order to get it call either engine.raw_connection() or engine.connect().connection:
from pandas.io import sql
#cnx = engine.connect().connection # option-1
cnx = engine.raw_connection() # option-2
xx = sql.read_frame("SELECT * FROM user", cnx)
cnx.close()
Use the MySQLdb module to create the connection. There is ongoing progress toward better SQL support, including sqlalchemy, but it's not ready yet.
If you are comfortable installing the development version of pandas, you might want to keep an eye on that linked issue and switch to using the development version of pandas as soon as it is merged. While pandas' SQL support is usable, there are some bugs around data types, missing values, etc., that are likely to come up if you use Pandas + SQL extensively.
This is an old question but still relevant apparently. So past 2018 the way to solve this is simply use the engine directly:
xx = sql.read_sql("SELECT * FROM user", engine)
(originally posted by Midnighter in a comment)
Related
I see lots of this question is about sqlite, but mine is to MySQL.
my entire script is like this:
df = pd.read_csv("df.csv")
engine = sqlalchemy.create_engine('mysql+mysqlconnector://{0}:{1}#{2}/{3}'.
format(config.user, config.passwd,
config.host, config.db))
df.to_sql('SQL_table', con=engine, if_exists='append', index=False)
Then it returns the error:
'Engine' object has no attribute 'cursor'
I googled, and followed some solutions, one of them is:
df = pd.read_csv("df.csv")
engine = sqlalchemy.create_engine('mysql+mysqlconnector://{0}:{1}#{2}/{3}'.
format(config.user, config.passwd,
config.host, config.db))
connection = engine.raw_connection()
df.to_sql('SQL_table', con=connection, if_exists='append', index=False)
Then the error changed to:
DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': Not all parameters were used in the SQL statement
I am using MySQL, not sqlite, i don't understand why it returns this error.
So basically, i think the solution is not working, and would anyone please tell me how to fix this problem, my SQLalchemy is 1.4.27
I have solved this, i reset my Mac, then come back to my VS Code, and start the notebook again, the problem is gone.
But before that, i tried also using command
reset
that can't do the trick. It has to be machine hard reset.
I have a Python database file, with MySQL Pooling setup like below
import mysql.connector
from mysql.connector import Error
from mysql.connector.connection import MySQLConnection
from mysql.connector import pooling
import pandas as pd
import datetime as dt
from contextlib import closing
#Outside of any function :
connection_pool = mysql.connector.pooling.MySQLConnectionPool(pool_name="database_pool",
pool_size=25,
pool_reset_session=True,
host='XXXXX',
database='XXXXX',
user='XXXXX',
password='XXXX')
In order to get a pooled connection I use the blow function located within the same file
def getDBConnection():
try:
connection_obj = connection_pool.get_connection()
cursor = connection_obj.cursor()
return connection_obj, cursor
except Error as e:
print(f"Error while connecting to MySQL using Connection pool : {e}")
Now lets say I want to preform a simple select function using a pooled connection (still within the same database file) - and then return the connection :
def dbGetDataHeadersForRunMenuBySubSet(strSubset):
connection_object, cursor = getDBConnection()
if connection_object.is_connected():
query = 'SELECT * FROM someTable'
cursor.execute(query)
#Now close the connection
closeDBConnection(connection_object, cursor)
code to attempt to close the Pool :
def closeDBConnection(connection_obj,cursor):
if connection_obj.is_connected():
connection_obj.close()
cursor.close()
However after 25 runs I get the error back saying
Error while connecting to MySQL using Connection pool : Failed getting connection; pool exhausted
Using the debugger I can see that the closeDBConnection is been run , and that it appears to hit every step with no errors.
So my question is :
Why am I running out of pools if I am closing them each time ?
All in all, I am actually looking to make a persistent connection , but in Python I cant find any realy examples on persistence , and all the examples I have looked at seem to point towards pooling. I am new (ish) to Python - so I have no issues here syaing that I know I have made a mistake somewhere.
Having played with this further :
adding "connection_object.close()" at the end of each individual function will free the connection_pool e.g
def dbGetDataHeadersForRunMenuBySubSet(strSubset):
connection_object, cursor = getDBConnection()
if connection_object.is_connected():
query = 'SELECT * FROM someTable'
cursor.execute(query)
#Now close the connection
#closeDBConnection(connection_object, cursor) <--- Not working for me
connection_object.close() <---- This WILL work instead. –
But the DB stuff is just so slow in comparrision to Excel mySQL Connecter (Excel is almost 3 times faster doing the same thing. I think this is because its easy to get a persistent connection in EXCEL - something which I cant do in python (I am new remember ;-) )
I am in the process of migrating databases from sqlite to mysql. Now that I've migrated the data to mysql, I'm not able to use my sqlalchemy code (in Python3) to access it in the new mysql db. I was under the impression that sqlalchemy syntax was database agnostic (i.e. the same syntax would work for accessing sqlite and mysql), but this appears not to be the case. So my question is: Is it absolutely required to use a DBAPI in addition to Sqlalchemy to read the data? Do I have to edit all of my sqlalchemy code to now read mysql?
The documentation says: The MySQL dialect uses mysql-python as the default DBAPI. There are many MySQL DBAPIs available, including MySQL-connector-python and OurSQL, which I think means that I DO need a DBAPI.
My old code with sqlite successfully worked like this with sqlite:
engine = create_engine('sqlite:///pmids_info.db')
def connection():
conn = engine.connect()
return conn
def load_tables():
metadata = MetaData(bind=engine) #init metadata. will be empty
metadata.reflect(engine) #retrieve db info for metadata (tables, columns, types)
inputPapers = Table('inputPapers', metadata)
return inputPapers
inputPapers = load_tables()
def db_inputPapers_retrieval(user_input):
result = engine.execute("select title, author, journal, pubdate, url from inputPapers where pmid = :0", [user_input])
for row in result:
title = row['title']
author = row['author']
journal = row['journal']
pubdate = row['pubdate']
url = row['url']
apa = str(author+' ('+pubdate+'). '+title+'. '+journal+'. Retrieved from '+url)
return apa
This worked fine and dandy. So then I tried to update it to work with the mysql db like this:
engine = create_engine('mysql://snarkshark#localhost/pmids_info')
At first when I tried to run my sample code like this, it complained because I didn't have MySqlDB. Some googling around informed me that MySqlDB does NOT work for Python 3. So then I tried pip installing pymysql and changing my engine statement to
engine = create_engine('mysql+pymysql://snarkshark#localhost/pmids_info')
which also ends up giving me various syntax errors when I try to adjust things.
So what I want to know, is if there is any way I can get my current syntax to work with mysql? Since the syntax is from sqlalchemy, I thought it would work perfectly for the exact same data in mysql that was previously in sqlite. Will I have to go through and update ALL of my db functions to use the syntax of the DBAPI?
This will sound like a dumb answer, but you'll need to change all the places where you're using database-specific behavior. SQLAlchemy does not guarantee that anything you do with it is portable across all backends. It leaks some abstractions on purpose to allow you to do things that are only available on certain backends. What you're doing is like using Python because it's cross-platform, then doing a bunch of os.fork()s everywhere, and then being surprised that it doesn't work on Windows.
For your specific case, at a minimum, you need to wrap all your raw SQL in text() so that you're not affected by the supported paramstyle of the DBAPI. However, there are still subtle differences between different dialects of SQL, so you'll need to use the SQLAlchemy SQL expression language instead of raw SQL if you want portability. After all that, you'll still need to be careful not to use backend-specific features in the SQL expression language.
I am going to build a site using Flask+MySQL+Flask-SQLAlchemy, however, after reading some tutorials, I have some questions:
Flask-SQLAlchemy can be imported in at least two ways:
http://pythonhosted.org/Flask-SQLAlchemy/quickstart.html
from flask.ext.sqlalchemy import SQLAlchemy
OR http://flask.pocoo.org/docs/patterns/sqlalchemy/
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
The first way seems much more convenient. Why Pocoo team choose to use the second way?
http://pythonhosted.org/Flask-SQLAlchemy/queries.html The insert examples here are too simple. How can I perform INSERT
IGNORE or INNER JOIN? If I want to write native SQL statements. How to do that with SQLAlchemy?
I need some good examples on MySQL+Flask-SQLAlchemy, while most example are SQLite+MySQL+Flask-SQLAlchemy.
I have been coding using MySQL+Flask-SQLAlchemy and i host my applications on pythonanywhere.com ,like what #Sean Vieira said ,your code will run on any Relational database the only thing you need to change is the connection string to the database of your liking ,e.g using Mysql ,this will be saved in a file called config.py [you can save it using any other name]
SQLALCHEMY_DATABASE_URI = 'mysql://username:password#localhost/yourdatabase'
SQLALCHEMY_POOL_RECYCLE = 280
SQLALCHEMY_POOL_SIZE = 20
SQLALCHEMY_TRACK_MODIFICATIONS = True
then in your main app ,you will import it like this
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
# Create an Instance of Flask
application = Flask(__name__)
# Include config from config.py
application.config.from_object('config')
application.secret_key = 'some_secret_key'
# Create an instance of SQLAlchemy
db = SQLAlchemy(application)
and all you need is to make sure your models correspond to a database table like the model below
class Isps(db.Model):
__tablename__ = "isps"
isp_id = db.Column('isp_id', db.Integer, primary_key=True)
isp_name = db.Column('isp_name', db.String(80), unique=True)
isp_description = db.Column('isp_description', db.String(180))
# services = db.relationship('Services', backref="isps", cascade="all, delete-orphan",lazy='dynamic')
def __init__(self, isp_name, isp_description):
self.isp_name = isp_name
self.isp_description = isp_description
def __repr__(self):
return '<Isps %r>' % self.isp_name
and you can then learn the power of SQLAlchemy to do optimised queries
Flask-SQLAlchemy was written by Armin (the creator of Flask) to make it simple to work with SQLAlchemy. The pattern described in Flask's documentation is what you would use if you did not choose to use the Flask-SQLAlchemy extension.
As for MySQL vs. SQLite, the whole point of the SQLAlchemy ORM is to make it possible (for the most part) to ignore what database you are running against. SomeModel.filter(SomeModel.column == 'value') will work the same, regardless of what database you are connecting to.
using:
Python 2.7.3
SQLAlchemy 0.7.8
PyODBC 3.0.3
I have implemented my own Dialect for the EXASolution DB using PyODBC as the underlying db driver. I need to make use of PyODBC's output_converter function to translate DECIMAL(x, 0) columns to integers/longs.
The following code snippet does the trick:
pyodbc = self.dbapi
dbapi_con = connection.connection
dbapi_version = dbapi_con.getinfo(pyodbc.SQL_DRIVER_VER)
(major, minor, patch) = [int(x) for x in dbapi_version]
if major >= 3:
dbapi_con.add_output_converter(pyodbc.SQL_DECIMAL, self.decimal2int)
I have placed this code snippet in the initialize(self, connection) method of
class EXADialect_pyodbc(PyODBCConnector, EXADialect):
Code gets called, and no exception is thrown, but this is a one time initialization. Later on, other connections are created. These connections are not passed through my initialization code.
Does anyone have a hint on how connection initialization works with SQLAlchemy, and where to place my code so that it gets called for every new connection created?
This is an old question, but something I hit recently, so an updated answer may help someone else along the way. In my case, I was trying to automatically downcase mssql UNIQUEIDENTIFIER columns (guids).
You can grab the raw connection (pyodbc) through the session or engine to do this:
engine = create_engine(connection_string)
make_session = sessionmaker(engine)
...
session = make_session()
session.connection().connection.add_output_converter(SQL_DECIMAL, decimal2int)
# or
connection = engine.connect().connection
connection.add_output_converter(SQL_DECIMAL, decimal2int)