using:
Python 2.7.3
SQLAlchemy 0.7.8
PyODBC 3.0.3
I have implemented my own Dialect for the EXASolution DB using PyODBC as the underlying db driver. I need to make use of PyODBC's output_converter function to translate DECIMAL(x, 0) columns to integers/longs.
The following code snippet does the trick:
pyodbc = self.dbapi
dbapi_con = connection.connection
dbapi_version = dbapi_con.getinfo(pyodbc.SQL_DRIVER_VER)
(major, minor, patch) = [int(x) for x in dbapi_version]
if major >= 3:
dbapi_con.add_output_converter(pyodbc.SQL_DECIMAL, self.decimal2int)
I have placed this code snippet in the initialize(self, connection) method of
class EXADialect_pyodbc(PyODBCConnector, EXADialect):
Code gets called, and no exception is thrown, but this is a one time initialization. Later on, other connections are created. These connections are not passed through my initialization code.
Does anyone have a hint on how connection initialization works with SQLAlchemy, and where to place my code so that it gets called for every new connection created?
This is an old question, but something I hit recently, so an updated answer may help someone else along the way. In my case, I was trying to automatically downcase mssql UNIQUEIDENTIFIER columns (guids).
You can grab the raw connection (pyodbc) through the session or engine to do this:
engine = create_engine(connection_string)
make_session = sessionmaker(engine)
...
session = make_session()
session.connection().connection.add_output_converter(SQL_DECIMAL, decimal2int)
# or
connection = engine.connect().connection
connection.add_output_converter(SQL_DECIMAL, decimal2int)
Related
I have a Python database file, with MySQL Pooling setup like below
import mysql.connector
from mysql.connector import Error
from mysql.connector.connection import MySQLConnection
from mysql.connector import pooling
import pandas as pd
import datetime as dt
from contextlib import closing
#Outside of any function :
connection_pool = mysql.connector.pooling.MySQLConnectionPool(pool_name="database_pool",
pool_size=25,
pool_reset_session=True,
host='XXXXX',
database='XXXXX',
user='XXXXX',
password='XXXX')
In order to get a pooled connection I use the blow function located within the same file
def getDBConnection():
try:
connection_obj = connection_pool.get_connection()
cursor = connection_obj.cursor()
return connection_obj, cursor
except Error as e:
print(f"Error while connecting to MySQL using Connection pool : {e}")
Now lets say I want to preform a simple select function using a pooled connection (still within the same database file) - and then return the connection :
def dbGetDataHeadersForRunMenuBySubSet(strSubset):
connection_object, cursor = getDBConnection()
if connection_object.is_connected():
query = 'SELECT * FROM someTable'
cursor.execute(query)
#Now close the connection
closeDBConnection(connection_object, cursor)
code to attempt to close the Pool :
def closeDBConnection(connection_obj,cursor):
if connection_obj.is_connected():
connection_obj.close()
cursor.close()
However after 25 runs I get the error back saying
Error while connecting to MySQL using Connection pool : Failed getting connection; pool exhausted
Using the debugger I can see that the closeDBConnection is been run , and that it appears to hit every step with no errors.
So my question is :
Why am I running out of pools if I am closing them each time ?
All in all, I am actually looking to make a persistent connection , but in Python I cant find any realy examples on persistence , and all the examples I have looked at seem to point towards pooling. I am new (ish) to Python - so I have no issues here syaing that I know I have made a mistake somewhere.
Having played with this further :
adding "connection_object.close()" at the end of each individual function will free the connection_pool e.g
def dbGetDataHeadersForRunMenuBySubSet(strSubset):
connection_object, cursor = getDBConnection()
if connection_object.is_connected():
query = 'SELECT * FROM someTable'
cursor.execute(query)
#Now close the connection
#closeDBConnection(connection_object, cursor) <--- Not working for me
connection_object.close() <---- This WILL work instead. –
But the DB stuff is just so slow in comparrision to Excel mySQL Connecter (Excel is almost 3 times faster doing the same thing. I think this is because its easy to get a persistent connection in EXCEL - something which I cant do in python (I am new remember ;-) )
I am in the process of migrating databases from sqlite to mysql. Now that I've migrated the data to mysql, I'm not able to use my sqlalchemy code (in Python3) to access it in the new mysql db. I was under the impression that sqlalchemy syntax was database agnostic (i.e. the same syntax would work for accessing sqlite and mysql), but this appears not to be the case. So my question is: Is it absolutely required to use a DBAPI in addition to Sqlalchemy to read the data? Do I have to edit all of my sqlalchemy code to now read mysql?
The documentation says: The MySQL dialect uses mysql-python as the default DBAPI. There are many MySQL DBAPIs available, including MySQL-connector-python and OurSQL, which I think means that I DO need a DBAPI.
My old code with sqlite successfully worked like this with sqlite:
engine = create_engine('sqlite:///pmids_info.db')
def connection():
conn = engine.connect()
return conn
def load_tables():
metadata = MetaData(bind=engine) #init metadata. will be empty
metadata.reflect(engine) #retrieve db info for metadata (tables, columns, types)
inputPapers = Table('inputPapers', metadata)
return inputPapers
inputPapers = load_tables()
def db_inputPapers_retrieval(user_input):
result = engine.execute("select title, author, journal, pubdate, url from inputPapers where pmid = :0", [user_input])
for row in result:
title = row['title']
author = row['author']
journal = row['journal']
pubdate = row['pubdate']
url = row['url']
apa = str(author+' ('+pubdate+'). '+title+'. '+journal+'. Retrieved from '+url)
return apa
This worked fine and dandy. So then I tried to update it to work with the mysql db like this:
engine = create_engine('mysql://snarkshark#localhost/pmids_info')
At first when I tried to run my sample code like this, it complained because I didn't have MySqlDB. Some googling around informed me that MySqlDB does NOT work for Python 3. So then I tried pip installing pymysql and changing my engine statement to
engine = create_engine('mysql+pymysql://snarkshark#localhost/pmids_info')
which also ends up giving me various syntax errors when I try to adjust things.
So what I want to know, is if there is any way I can get my current syntax to work with mysql? Since the syntax is from sqlalchemy, I thought it would work perfectly for the exact same data in mysql that was previously in sqlite. Will I have to go through and update ALL of my db functions to use the syntax of the DBAPI?
This will sound like a dumb answer, but you'll need to change all the places where you're using database-specific behavior. SQLAlchemy does not guarantee that anything you do with it is portable across all backends. It leaks some abstractions on purpose to allow you to do things that are only available on certain backends. What you're doing is like using Python because it's cross-platform, then doing a bunch of os.fork()s everywhere, and then being surprised that it doesn't work on Windows.
For your specific case, at a minimum, you need to wrap all your raw SQL in text() so that you're not affected by the supported paramstyle of the DBAPI. However, there are still subtle differences between different dialects of SQL, so you'll need to use the SQLAlchemy SQL expression language instead of raw SQL if you want portability. After all that, you'll still need to be careful not to use backend-specific features in the SQL expression language.
I stumbled upon the following:
def save_formset(self, request, form, formset, change):
instances = formset.save(commit=False)
bargain_id = 0
total_price = Decimal(0)
for instance in instances:
if isinstance(instance, BargainProduct):
total_price += instance.quantity * instance.product.price
bargain_id = instance.id
instance.save()
updateTotal = Bargain.objects.get(id=bargain_id)
updateTotal.total_price = total_price - updateTotal.discount_price
updateTotal.save()
This code is working for me on my local MySQL setup, however, on my live test enviroment running on SQLite3* I get the "Bargain matching query does not exist." error..
I am figuring this is due to a different hierarchy of saving the instances on SQLite.. however it seems they run(and should) act the same..?
*I cannot recompile MySQL with python support on my liveserver atm so thats a no go
Looking at the code, if you have no instances coming out of the formset.save(), bargain_id will be 0 when it gets down to the Bargain.objects.get(id=bargain_id) line, since it will skip over the for loop. If it is 0, I'm guessing it will fail with the error you are seeing.
You might want to check to see if the values are getting stored correctly in the database during your formset.save() and it is returning something back to instances.
This line is giving the error:
updateTotal = Bargain.objects.get(id=bargain_id)
which most probably is because of this line:
instances = formset.save(commit=False)
Did you define a save() method for the formset? Because it doesn't seen to have one built-in. You save it by accessing what formset.cleaned_data returns as the django docs say.
edit: I correct myself, it actually has a save() method based on this page.
I've been looking at this same issue. It is saving the data to the database, and the formset is filled. The problem is that the save on instances = formset.save(commit=False) doesn't return a value. When I look at the built-in save method, it should give back the saved data.
Another weird thing about this, is that it seems to work on my friends MySQL backend, but not on his SQLITE3 backend. Next to that it doesn't work on my MySQL backend.
Local loop returns these print outs (on MySQL).. on sqlite3 it fails with a does not excist on the query
('Formset: ', <django.forms.formsets.BargainProductFormFormSet object at 0x101fe3790>)
('Instances: ', [<BargainProduct: BargainProduct object>])
[18/Apr/2011 14:46:20] "POST /admin/shop/deal/add/ HTTP/1.1" 302 0
Got a Zend Framework web application using Doctrine 1.2 connecting to a MYSQL 5.1 server.
Most data needs to be entered and displayed in local timezone. So following the advice here, I'd like (I think) to set PHP and MySQL to use UTC and then I'll handle the conversion to local time for display and to UTC prior to insert/update. [If this is totally goofy, I'm happy to to hear a better approach.]
So, how do I tell Doctrine to set the MySQL session to UTC? Essentially, how do I tell Doctrine to issue the MySQL command SET SESSION time_zone = 'UTC'; when it opens the connection?
Thanks in advance!
It appears that this can be done by attaching to the Doctrine_Connection object an Doctrine_EventListener with a postConnect() method.
Doctrine ORM for PHP - Creating a New Listener
Something like:
class Kwis_Doctrine_EventListener_Timezone extends Doctrine_EventListener
{
protected $_timezone;
public function __construct($timezone = 'UTC')
{
$timezone = (string) $timezone;
$this->_timezone = $timezone;
}
public function postConnect(Doctrine_Event $event)
{
$conn = $event->getInvoker();
$conn->execute(sprintf("SET session time_zone = '%s';", $this->_timezone));
}
}
[Using sprintf() here is probably amateurish, but I couldn't figure out how to do a proper parameter bind. (embarrassed smiley).]
Then in my app's Bootstrap.php, I have something like:
protected function _initDoctrine()
{
// various operations relating to autoloading
// ..
$doctrineConfig = $this->getOption('doctrine');
$manager = Doctrine_Manager::getInstance();
// various boostrapping operations on the manager
// ..
$conn = Doctrine_Manager::connection($doctrineConfig['dsn'], 'doctrine');
// various boostrapping operations on the connection
// ..
// Here's the good stuff: Add the EventListener
$conn->addListener(new Kwis_Doctrine_EventListener_Timezone());
return $conn;
}
[One slightly off-topic note: on my local development machine, I kept running into a MySQL error in which no timezone I entered seemed to be accepted. Turns out that the standard MySQL install creates all the timezone tables in the mysql database, but doesn't actually populate them; you need to populate them separately. More info at MySQL site.]
I've inherited a Pylons app that uses SQLAlchemy. I know nothing about SQLAlchemy and very little about Pylons :)
I need to run some raw SQL from within the app. The SQLAlchemy currently seems to be working in the following way (example code):
import myapp.model as model
model.Session.query(model.KeyValue) # existing code
.join(model.Key)
.filter(model.Key.name == name)
).count() == 0, name
How do I get it to run raw SQL? I see that I need an execute() statement, but how exactly do I run it? The following both fail:
model.Session.execute('create table hello_world;')
model.Connection.execute("""
create table hello_world;
""")
What's the magic invocation? There's no reference to a Connection object in the existing code, and I'm not sure how to create one.
You can obtain connection that is used by Session by using its connection method:
connection = model.Session.connection()
Then you can issue your query:
connection.execute('create table hello_world;')
Note that in Pylons model.Session is not a sqlalchemy.orm.session.Session class. It's an instance of sqlalchemy.orm.scoping.ScopedSession. That's how it's created in model.meta module:
Session = scoped_session(sessionmaker())
My first impulse is to recommend trying the execute() method of an instance of Connection, instead of the execute() method of the class itself as your example code suggests that you're doing.
Are you working off of the Pylons Book examples ?