SQLAlchemy and adodbapi Database connection error - sqlalchemy

I'm attempting to connect to a mssql SQLExpress 2012 database using sqlalchemy 0.7.8 and adodapi 2.4.2.2 on IronPython 2.7.3
I am able to create a sqlalchemy engine, however when a query is made I get :
"TypeError: 'NoneType' object is unsubscriptable"
TraceBack:
Traceback (most recent call last):
File "C:\Program Files (x86)\IronPython 2.7\Lib\site-packages\SQLAlchemy-0.7.8-py2.7.egg\sqlalchemy\engine\base.py", line 878, in __init__
File "C:\Program Files (x86)\IronPython 2.7\Lib\site-packages\SQLAlchemy-0.7.8-py2.7.egg\sqlalchemy\engine\base.py", line 2558, in raw_connection
File "C:\Program Files (x86)\IronPython 2.7\Lib\site-packages\SQLAlchemy-0.7.8-py2.7.egg\sqlalchemy\pool.py", line 183, in unique_connection
File "<string>", line 9, in <module>
File "C:\Program Files (x86)\IronPython 2.7\Lib\site-packages\SQLAlchemy-0.7.8-py2.7.egg\sqlalchemy\engine\base.py", line 2472, in connect
TypeError: 'NoneType' object is unsubscriptable
Code being used:
def conn():
return adodbapi.connect('Provider=SQLOLEDB; Data Source=SERVER\SQLEXPRESS;
Initial Catalog=db; User ID=user; Password=pass;')
engine = create_engine('mssql+adodbapi:///', creator=conn,
echo = True, module=adodbapi)
adodbapi seems to work fine on it's own, ie. i can create a connection and then use a cursor to query without any problems, it seems to be something in sqlalchemy.
Anyone any ideas?

And we have a workaround:
import adodbapi
from sqlalchemy.engine import create_engine
from sqlalchemy.orm import sessionmaker
import sqlalchemy.pool as pool
def connect():
return adodbapi.connect('Provider=SQLOLEDB.1;Data Source=mypcname\SQLEXPRESS;\
Initial Catalog=dbname;User ID=user; Password=pass;')
mypool = pool.QueuePool(connect)
conn = mypool.connect()
curs = conn.cursor()
curs.execute('select 1') #anything that forces open the connection
engine = create_engine('mssql+adodbapi://', module=adodbapi, pool = mypool)
Session = sessionmaker()
Session.configure(bind=engine)
sess = Session()
With this my session object works as normal.
I'm probably not using the adodbapi dialect as intended by whoever made it, but I can't find any documentation, so this is what I've gone with for now.

Pretty sure adodbapi doesn't work with SQLAlchemy.
The adodbapi dialect is not implemented for 0.6 at this time.
Scroll to the very bottom, (this is 0.7x documentation), I also checked 0.8 documentation and it says the same thing.
Sounds like you'll have to change which driver you're using.

I use sqlalcmy to connect to a postgresql database using the psycopg2. I am not sure, but by reading the documentation, i think you need to download the pyodbc, it seems to be better supported than adodbapi. Once you have installed it, try the following statement to create the engine
engine = create_engine(mssql+pyodbc://user:pass#host/db)
Or you can check out different ways of writing the connection string here.

Related

How do I connect Airflow to SQLite locally?

I'm trying to try out Airflow for the very first time and I'm trying to connect it to a local SQLite database. But I can't seem to get my head around on how to actually do it.
I've read up on Airflow's document, Set my executor to LocalExecutor and set up my sql_alchemy_conn to sqlite:////home/myName/Programs/sqlite3/DatabaseName.db but it doesn't seem to work as it throws an
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 21, in <module>
from airflow import configuration
File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py", line 35, in <module>
from airflow import configuration as conf
File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", line 520, in <module>
conf.read(AIRFLOW_CONFIG)
File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", line 283, in read
self._validate()
File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", line 169, in _validate
self.get('core', 'executor')))
airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor
error when I tried to run airflow initdb. I tried to google around and tried vipul sharma's solution found here and set the value of my sql_alchemy_conn to mysql://:#localhost:3306/ but it still doesn't work as it throws an
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'myName'#'localhost' (using password: NO)")
error. I know that the answer should be really simple but I really don't understand how to so I hope you can guide me through on what to do/read.
Use SequentialExecutor
"This executor will only run one task instance at a time, can be used for debugging. It is also the only executor that can be used with sqlite since sqlite doesn’t support multiple connections." airflow documentation
You just didn't need to change to LocalExecutor, change it back to SequentialExecutor, change sql_alchemy_conn to point to sqlite:////home/myName/Programs/sqlite3/DatabaseName.db and stop airflow services (webserver, scheduler).
Execute airflow initdb then start up the services again.
Hopefully that works.

Spark Read.json cant find file

Hey I all I have 1 Master and 1 Slave Node Standalone Spark Cluster on AWS. I have a folder my home directory called ~/Notebooks. This is were I launch jupyter notebooks and connect jupyter in my browser. I also have a file in there called people.json (simple json file).
I try running this code
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
conf = SparkConf().setAppName('Practice').setMaster('spark://ip-172-31-2-186:7077')
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
df = sqlContext.read.json("people.json")
I get this error when i run that last line. I don't get it the file is right there... Any Ideas?-
Py4JJavaError: An error occurred while calling o238.json.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 4.0 failed 4 times, most recent failure: Lost task 1.3 in stage 4.0 (TID 37, ip-172-31-7-160.us-west-2.compute.internal): java.io.FileNotFoundException: File file:/home/ubuntu/Notebooks/people.json does not exist
Make sure the file is available on the worker nodes. Best way is to use a shared files system (NFS, HDFS). Read External Datasets documentation

MySQL Connector/Python 1.0.7 on OSX 10.8 not getting a cursor

Here's a skeleton of my Python code:
import mysql.connector
sql = mysql.connector.connect(user='user', password='pwd',
host='127.0.0.1', database='my_db')
cursor = sql.cursor()
Yields this mysterious error message:
Traceback (most recent call last):
File "pyscript_v2.py", line 36, in
cursor = sql.cursor()
file "/Library/Python/2.7/site-packages/mysql/connector/connection.py", line 1063, in cursor raise errors.OperationalError("MySQL Connection not available.")
mysql.connector.errors.OperationalError: MySQL Connection not available.
My Google-fu is failing to dig anything up, am I missing something in setting up my MySQL DB connection in Python?
I'm running Python 2.7.2 on OS X 10.8.2, and am using MySQL Connector/Python 1.0.7.
Following link has explanation and possible solution.
http://bugs.mysql.com/bug.php?id=67649

web2py doesn't connect to mysql

I installed web2py as source and wanted to use DAL without the rest of the framework.
But DAL does not connect to mysql:
>>> DAL('mysql://user1:user1#localhost/test_rma')
...
RuntimeError: Failure to connect, tried 5 times:
'NoneType' object has no attribute 'connect'
Whereas MySQLdb can connect to the database with the same credentials:
>>> import MySQLdb
>>> db = MySQLdb.connect(host='localhost', user='user1', passwd='user1', db='test_rma')
A similar problem with MsSQL was solved by explicitly setting the driver object. I tried the same solution:
>>> from gluon.dal import MySQLAdapter
>>> print MySQLAdapter.driver
None
>>> driver = globals().get('MySQLdb',None)
>>> print MySQLAdapter.driver
None
But still the driver is None.
Ok, I found the solution of the problem. I had to write:
MySQLAdapter.driver = globals().get('MySQLdb',None)
instead of
driver = globals().get('MySQLdb',None)
I misread that line in the original question.

Using SQL Alchemy and pyodbc with IronPython 2.6.1

I'm using IronPython and the clr module to retrieve SQL Server information via SMO. I'd like to retrieve/store this data in a SQL Server database using SQL Alchemy, but am having some trouble loading the pyodbc module.
Here's the setup:
IronPython 2.6.1 (installed at D:\Program Files\IronPython)
CPython 2.6.5 (installed at D:\Python26)
SQL Alchemy 0.6.1 (installed at D:\Python26\Lib\site-packages\sqlalchemy)
pyodbc 2.1.7 (installed at D:\Python26\Lib\site-packages)
I have these entries in the IronPython site.py to import CPython standard and third-party libraries:
# Add CPython standard libs and DLLs
import sys
sys.path.append(r"D:\Python26\Lib")
sys.path.append(r"D:\Python26\DLLs")
sys.path.append(r"D:\Python26\lib-tk")
sys.path.append(r"D:\Python26")
# Add CPython third-party libs
sys.path.append(r"D:\Python26\Lib\site-packages")
# sqlite3
sys.path.append(r"D:\Python26\Lib\sqlite3")
# Add SQL Server SMO
sys.path.append(r"D:\Program Files\Microsoft SQL Server\100\SDK\Assemblies")
import clr
clr.AddReferenceToFile('Microsoft.SqlServer.Smo.dll')
clr.AddReferenceToFile('Microsoft.SqlServer.SqlEnum.dll')
clr.AddReferenceToFile('Microsoft.SqlServer.ConnectionInfo.dll')
SQL Alchemy imports OK in IronPython, put I receive this error message when trying to connect to SQL Server:
IronPython 2.6.1 (2.6.10920.0) on .NET 2.0.50727.3607
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlalchemy
>>> e = sqlalchemy.MetaData("mssql://")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "D:\Python26\Lib\site-packages\sqlalchemy\schema.py", line 1780, in __init__
File "D:\Python26\Lib\site-packages\sqlalchemy\schema.py", line 1828, in _bind_to
File "D:\Python26\Lib\site-packages\sqlalchemy\engine\__init__.py", line 241, in create_engine
File "D:\Python26\Lib\site-packages\sqlalchemy\engine\strategies.py", line 60, in create
File "D:\Python26\Lib\site-packages\sqlalchemy\connectors\pyodbc.py", line 29, in dbapi
ImportError: No module named pyodbc
This code works just fine in CPython, but it looks like the pyodbc module isn't accessible from IronPython.
Any suggestions? I realize that this may not be the best way to approach the problem, so I'm open to tackling this a different way. Just wanted to get some experience with using SQL Alchemy and pyodbc.
its very likely that pyodbc is not compatible with IronPython, as it was designed for usage with cPython.
IronPython certainly has some kind of ODBC (actually, ADO.net seems like where its at) compatibility built into it, but a DBAPI would be the most direct way to get SQLAlchemy working with it.
So here's some MS-specific non-DBAPI example: http://www.ironpython.info/index.php/Accessing_SQL_Server
someone talking about DBAPI in 2006: http://hex-dump.blogspot.com/2006/10/ironpython-and-adonet-part-2.html
something a little more recent: http://bitbucket.org/jdhardy/adonet-dbapi/
It says something that MS pours however much money into IronPython but zero into a compliant DBAPI driver.
You could try using SQLAlchemy's adodbapi support instead; the latest version of adodbapi (2.3.0) supports IronPython.
You should only have to make sure the adodbapi package is on sys.path, and then use 'mssql+adodbapi://' instead of 'mssql://' in your connection string.
adodbapi seems the way to go, but here's a snippet from adodbapi.py that ships with SQL Alchemy under the dialects folder
"""
The adodbapi dialect is not implemented for 0.6 at this time.
"""
SQLAlchemy can not directly run under IronPython, because pyodbc currently is not compatible with IronPython.
However, you can use pypyodbc under IronPython as a dbi-2.0 complaint library, which is similar to pyodbc,and enables running sqlalchemy under Ironpython, this How-to describes the 4 steps to enable it.
Disclaimer: I'm the maintianer of pypyodbc.