Using SQL Alchemy and pyodbc with IronPython 2.6.1 - sqlalchemy

I'm using IronPython and the clr module to retrieve SQL Server information via SMO. I'd like to retrieve/store this data in a SQL Server database using SQL Alchemy, but am having some trouble loading the pyodbc module.
Here's the setup:
IronPython 2.6.1 (installed at D:\Program Files\IronPython)
CPython 2.6.5 (installed at D:\Python26)
SQL Alchemy 0.6.1 (installed at D:\Python26\Lib\site-packages\sqlalchemy)
pyodbc 2.1.7 (installed at D:\Python26\Lib\site-packages)
I have these entries in the IronPython site.py to import CPython standard and third-party libraries:
# Add CPython standard libs and DLLs
import sys
sys.path.append(r"D:\Python26\Lib")
sys.path.append(r"D:\Python26\DLLs")
sys.path.append(r"D:\Python26\lib-tk")
sys.path.append(r"D:\Python26")
# Add CPython third-party libs
sys.path.append(r"D:\Python26\Lib\site-packages")
# sqlite3
sys.path.append(r"D:\Python26\Lib\sqlite3")
# Add SQL Server SMO
sys.path.append(r"D:\Program Files\Microsoft SQL Server\100\SDK\Assemblies")
import clr
clr.AddReferenceToFile('Microsoft.SqlServer.Smo.dll')
clr.AddReferenceToFile('Microsoft.SqlServer.SqlEnum.dll')
clr.AddReferenceToFile('Microsoft.SqlServer.ConnectionInfo.dll')
SQL Alchemy imports OK in IronPython, put I receive this error message when trying to connect to SQL Server:
IronPython 2.6.1 (2.6.10920.0) on .NET 2.0.50727.3607
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlalchemy
>>> e = sqlalchemy.MetaData("mssql://")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "D:\Python26\Lib\site-packages\sqlalchemy\schema.py", line 1780, in __init__
File "D:\Python26\Lib\site-packages\sqlalchemy\schema.py", line 1828, in _bind_to
File "D:\Python26\Lib\site-packages\sqlalchemy\engine\__init__.py", line 241, in create_engine
File "D:\Python26\Lib\site-packages\sqlalchemy\engine\strategies.py", line 60, in create
File "D:\Python26\Lib\site-packages\sqlalchemy\connectors\pyodbc.py", line 29, in dbapi
ImportError: No module named pyodbc
This code works just fine in CPython, but it looks like the pyodbc module isn't accessible from IronPython.
Any suggestions? I realize that this may not be the best way to approach the problem, so I'm open to tackling this a different way. Just wanted to get some experience with using SQL Alchemy and pyodbc.

its very likely that pyodbc is not compatible with IronPython, as it was designed for usage with cPython.
IronPython certainly has some kind of ODBC (actually, ADO.net seems like where its at) compatibility built into it, but a DBAPI would be the most direct way to get SQLAlchemy working with it.
So here's some MS-specific non-DBAPI example: http://www.ironpython.info/index.php/Accessing_SQL_Server
someone talking about DBAPI in 2006: http://hex-dump.blogspot.com/2006/10/ironpython-and-adonet-part-2.html
something a little more recent: http://bitbucket.org/jdhardy/adonet-dbapi/
It says something that MS pours however much money into IronPython but zero into a compliant DBAPI driver.

You could try using SQLAlchemy's adodbapi support instead; the latest version of adodbapi (2.3.0) supports IronPython.
You should only have to make sure the adodbapi package is on sys.path, and then use 'mssql+adodbapi://' instead of 'mssql://' in your connection string.

adodbapi seems the way to go, but here's a snippet from adodbapi.py that ships with SQL Alchemy under the dialects folder
"""
The adodbapi dialect is not implemented for 0.6 at this time.
"""

SQLAlchemy can not directly run under IronPython, because pyodbc currently is not compatible with IronPython.
However, you can use pypyodbc under IronPython as a dbi-2.0 complaint library, which is similar to pyodbc,and enables running sqlalchemy under Ironpython, this How-to describes the 4 steps to enable it.
Disclaimer: I'm the maintianer of pypyodbc.

Related

Broken MySql driver for PySide2 on Windows?

I've been forced to try to port to PySide2 from PySide because I'm developing code (on Windows 7) used inside Maya and Maya 2017 has moved to PySide2.
PySide2 ships with Maya but only comes with the driver to talk to sqlite DBs. So I need to tell Qt about a new DIR that contains the MySql driver, like this:
from PySide2 import QtWidgets
QtWidgets.QApplication.addLibraryPath("C:\temp\plugins")
This DIR contains a sub sqldrivers DIR which contains the driver. This works- when I call QtSql.QSqlDatabase.drivers() the new one is there so I've got the right driver and it's being picked up.
However when I create a new DB and call open:
db = QtSql.QSqlDatabase.addDatabase("QMYSQL3")
db.setDatabaseName("my_db")
db.setHostName("host")
db.setUserName("user")
db.setPassword("password")
db.open()
the open command returns False and the DB is not accessible. I call:
db.lastError()
and get:
# Result: <PySide2.QtSql.QSqlError("", "Driver not loaded", "Driver not loaded") at 0x0000000075410C08> #
NB: we successfully did the exact same thing with the previous PySide version.
Other threads mention similar things and it turned out to be missing the actual libmysql.dll, which is the actual MySql DLL and can result in the same error if missing. However I definitely have this on my system and it's definitely in the PATH. It's actually the same file that is being successfully picked up through PySide. But the PySide2 driver just wont open a connection to the DB.
I suspect maybe a dodgy driver but wondered if anyone has any experience with this or any advice?
I had the same error, but with postgres.
I solved by placing the libpq.dll connector library next to my script.py.
Then for mysql it would be libmysql.dll.
In my case I had to build the libpq.dll from the source of postgres 9.6
It seems that with PySide2 it is not necessary to use:
QtWidgets.QApplication.addLibraryPath ("C: \ path_to\ plugins")
using:
print (PySide2.QtCore.QLibraryInfo.location (PySide2.QtCore.QLibraryInfo.PluginsPath)), I could see that the path was correct.
a greeting

calling legacy c functions from python

I want to call legacy c third party functions from python.
I created a C api to make the function calls simpler.
In my python file
I tried to import the *.so for the api which links with the legacy code library.
But I kept seeing a libxxx ( the legacy c library) not found error.
I tried to import both ans still saw the same error.
I then tried using ctypes, but get the error ctypes not found.
Any suggestions ?
# python
Python 2.4.2 (#1, Apr 20 2012, 19:31:49)
[GCC 4.1.2 20070115 (SUSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ImportError: No module named ctypes
>>> quit
ctypes only made it into the standard library in version 2.5, so 2.4 does not have it. I've also heard that some people exclude it when they build Python for a distribution or for embedding in a specific application, as you can easily crash your process using it, or even corrupt the host application's heap. But I have never encountered this myself, and I doubt a full installation does this.
As for not finding your C library, we'll need more details on that.

SQLAlchemy and adodbapi Database connection error

I'm attempting to connect to a mssql SQLExpress 2012 database using sqlalchemy 0.7.8 and adodapi 2.4.2.2 on IronPython 2.7.3
I am able to create a sqlalchemy engine, however when a query is made I get :
"TypeError: 'NoneType' object is unsubscriptable"
TraceBack:
Traceback (most recent call last):
File "C:\Program Files (x86)\IronPython 2.7\Lib\site-packages\SQLAlchemy-0.7.8-py2.7.egg\sqlalchemy\engine\base.py", line 878, in __init__
File "C:\Program Files (x86)\IronPython 2.7\Lib\site-packages\SQLAlchemy-0.7.8-py2.7.egg\sqlalchemy\engine\base.py", line 2558, in raw_connection
File "C:\Program Files (x86)\IronPython 2.7\Lib\site-packages\SQLAlchemy-0.7.8-py2.7.egg\sqlalchemy\pool.py", line 183, in unique_connection
File "<string>", line 9, in <module>
File "C:\Program Files (x86)\IronPython 2.7\Lib\site-packages\SQLAlchemy-0.7.8-py2.7.egg\sqlalchemy\engine\base.py", line 2472, in connect
TypeError: 'NoneType' object is unsubscriptable
Code being used:
def conn():
return adodbapi.connect('Provider=SQLOLEDB; Data Source=SERVER\SQLEXPRESS;
Initial Catalog=db; User ID=user; Password=pass;')
engine = create_engine('mssql+adodbapi:///', creator=conn,
echo = True, module=adodbapi)
adodbapi seems to work fine on it's own, ie. i can create a connection and then use a cursor to query without any problems, it seems to be something in sqlalchemy.
Anyone any ideas?
And we have a workaround:
import adodbapi
from sqlalchemy.engine import create_engine
from sqlalchemy.orm import sessionmaker
import sqlalchemy.pool as pool
def connect():
return adodbapi.connect('Provider=SQLOLEDB.1;Data Source=mypcname\SQLEXPRESS;\
Initial Catalog=dbname;User ID=user; Password=pass;')
mypool = pool.QueuePool(connect)
conn = mypool.connect()
curs = conn.cursor()
curs.execute('select 1') #anything that forces open the connection
engine = create_engine('mssql+adodbapi://', module=adodbapi, pool = mypool)
Session = sessionmaker()
Session.configure(bind=engine)
sess = Session()
With this my session object works as normal.
I'm probably not using the adodbapi dialect as intended by whoever made it, but I can't find any documentation, so this is what I've gone with for now.
Pretty sure adodbapi doesn't work with SQLAlchemy.
The adodbapi dialect is not implemented for 0.6 at this time.
Scroll to the very bottom, (this is 0.7x documentation), I also checked 0.8 documentation and it says the same thing.
Sounds like you'll have to change which driver you're using.
I use sqlalcmy to connect to a postgresql database using the psycopg2. I am not sure, but by reading the documentation, i think you need to download the pyodbc, it seems to be better supported than adodbapi. Once you have installed it, try the following statement to create the engine
engine = create_engine(mssql+pyodbc://user:pass#host/db)
Or you can check out different ways of writing the connection string here.

Using MySQL with dev_appserver (Google App Engine) and Google SQL Service

I am using Google App Engine and Google SQL Service, and would like to use the option
dev_appserver.py --mysql_user=username myapp
in order to use a local MySQL database for development purposes, and SQL Service in prod environment.
I have MySQLdb installed and working:
/usr/bin>>python
Python 2.7.2 (v2.7.2:8527427914a2, Jun 11 2011, 15:22:34)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>>
However, when I run the dev_appserver command, I get the following error:
zipimporter('/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQL_python-1.2.3-py2.7-macosx-10.6-intel.egg', 'MySQLdb/')
ERROR 2011-09-12 09:34:53,541 rdbms_mysqldb.py:90] The rdbms API is not available because the MySQLdb library could not be loaded.
I have verified that the MySQL_python-1.2.3-py2.7-macosx-10.6-intel.egg is in the location (/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/) and that it is accessible.
Any idea what might be causing this error?
Full stacktrace provided below for information:
/Users/eddieboyd>>dev_appserver.py --debug --mysql_user=mysql eddiehelloworld
Warning: You are using a Python runtime (2.7) that is more recent than the production runtime environment (2.5). Your application may use features that are not available in the production environment and may not work correctly when deployed to production.
INFO 2011-09-12 09:34:53,290 appengine_rpc.py:159] Server: appengine.google.com
INFO 2011-09-12 09:34:53,304 appcfg.py:449] Checking for updates to the SDK.
DEBUG 2011-09-12 09:34:53,305 appengine_rpc.py:364] Sending HTTPS request:
POST /api/updatecheck?release=1.5.3&timestamp=1311108376&api_versions=%5B%271%27%5D HTTPS/1.1
Host: appengine.google.com
X-appcfg-api-version: 1
Content-type: application/octet-stream
User-agent: appcfg_py/1.5.3 Darwin/10.8.0 Python/2.7.2.final.0
INFO 2011-09-12 09:34:53,465 appcfg.py:466] The SDK is up to date.
WARNING 2011-09-12 09:34:53,465 datastore_file_stub.py:512] Could not read datastore data from /var/folders/++/++71vE++6+0++4RjPqRgNE+0Eyo/-Tmp-/dev_appserver.datastore
INFO 2011-09-12 09:34:53,478 py_zipimport.py:148] zipimporter('/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQL_python-1.2.3-py2.7-macosx-10.6-intel.egg', 'MySQLdb/')
ERROR 2011-09-12 09:34:53,541 rdbms_mysqldb.py:90] The rdbms API is not available because the MySQLdb library could not be loaded.
ERROR 2011-09-12 09:34:53,541 dev_appserver_main.py:638] <type 'exceptions.NotImplementedError'>: Unable to find the MySQLdb library. Please see the SDK documentation for installation instructions.
DEBUG 2011-09-12 09:34:53,543 dev_appserver_main.py:640] Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver_main.py", line 635, in main
dev_appserver.SetupStubs(appinfo.application, **option_dict)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 4668, in SetupStubs
rdbms_mysqldb.connect(database='')
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/rdbms_mysqldb.py", line 96, in connect
'Unable to find the MySQLdb library. Please see the SDK '
NotImplementedError: Unable to find the MySQLdb library. Please see the SDK documentation for installation instructions.
Poor man's solution.
I added
"import MySQLdb"
in dev_appserver.py.
It works now.
Ensure to use the latest version of the AppEngine SDK which currently is the version 1.6.2.
After that install the package for python mysql support: python-mysqldb.
Maybe your system does not have the MySQL module required. On Fedora 16 64 bits, I solved this by installing the MySQL python module:
yum install MySQL-python

Why does Jython refuse to find my Java package?

I know it's something silly, but for some reason Jython refuses to find javax.swing. I'm using Java 1.6.0_11. This is my start-up script:
#echo off
"%JAVA_HOME%\bin\java" -Xmx1024M -classpath ".;c:\Projects\Jython2.5.1\jython.jar" org.python.util.jython
My output looks like:
Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.6.0_10
Type "help", "copyright", "credits" or "license" for more information.
>>> import javax.swing
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named swing
>>> import javax
>>> dir(javax)
['__name__']
>>>
Most likely Jython is not scanning your packages. On startup, Jython tries to go through the jars and class files on its path and scan for Java packages. This is necessary because there is no way to look for Java packages by reflection. Package scanning can be deliberately turned off, or you could lack write privileges where it wants to write the cached information out see http://wiki.python.org/jython/PackageScanning for more. The best way to import Java classes is to do so explicitly class by class, like so:
from javax.swing import JFrame
This method should always work, even if package scanning is off or otherwise unable to work, and is the recommended approach (though it can be a bit tedious). If you do want to import packages (or if you want to do "from javax.swing import *" which also depends on package scanning - but is discouraged) you will need to figure out why your package scanning isn't working.
I had similar issues, and it turns out that since the standalone Jython dist does not support caching, it also does not support the "import *" approach. This is not clearly documented anywhere in the official Jython docs, but I concluded this based on a number of different bug reports:
https://groups.google.com/forum/?hl=en&fromgroups=#!topic/robotframework-users/6ipB0DYJkvU
http://bugs.jython.org/issue1778514
http://bugs.jython.org/issue1422
http://bugs.jython.org/issue1692579
Notable from that last link:
So as Oti noted, in standalone you must do full imports to succeed.
To fix your issue, use the non-standalone standard jython.jar generated by installing jython using the 'Standard' option.
If you wanted to package and distribute jython.jar with your application, in case a user does not have Jython installed, then you will also need to copy/pase the complete "Lib" folder from the jython installation directory into whichever location you end up placing jython.jar. This enables access to the python stdlib which is not included in the standard jar file.
UPDATE:
After playing around more, I think I have a fix to enable "import *" type imports even when using the standalone jar. All that needs to be done is to enable caching!
You can do this by either adding the following options to the jvm when running jython:
-Dpython.cachedir.skip=false -Dpython.cachedir=DESIRED CACHE PATH
(Note that the second argument is optional, and if left blank, a default value will be used)
If you are having an issue running the InteractiveConsole embedded in an app (which is what my problem was) you can add these properties before initializing the console:
Properties props = new Properties();
props.put("python.cachedir.skip", "false");
props.put("python.cachedir", "DESIRED CACHE PATH"); // again, this option is optional
InteractiveConsole.initialize(System.getProperties(), props, new String[0]);
I'm using Java 1.6.0_11
No, you're using
[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.6.0_10
What happens if you delete the cachedir from the Jython distribution directory, and try again?
Also, why are you explicitly setting the classpath that way? Why not simply
java -jar jython.jar
?