Currently:
SQLAlchemy installed and working (or at least import v0.8.0b2)
Mysql (v5.5.16)
Distribute (0.6.34)
Oracle mysql-python connector
Python 3.2
Windows 7 32/64 (note that I installed Python 32bits)
The problem is that MySQLdb or Oursql is required and I didn't managed to get any of them working.
Found this but didn't manage to get it working neither.
Edit: If you are aware of an other orm that works with Python3, I'm interested.
I was successful in getting Oracle's MySQL connector for python working with SQLAlchemy on Python 3.3. Your connection string needs to start with "mysql+mysqlconnector://...". After I changed my connection string everything (well, simple things) started working.
The MySQL connector docs can be found here: https://dev.mysql.com/doc/connector-python/en/
The package is up on PyPi: https://pypi.org/project/mysql-connector-python/
Here are the SQLAlchemy docs about using the Python connector: http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#module-sqlalchemy.dialects.mysql.mysqlconnector
For others who arrive here, this should do it:
pip install mysql-connector==2.1.4 # version avoids Protobuf error
URI = 'mysql+mysqlconnector://$USER:$PASS#$HOST/$DB'
I tried Oracle's connection, as suggested by #Brad Campbell, but unfortunately it was extremely slow, much slower than the "real" MySQL-Python connection I had been using with SQLAlchemy on Python 2.
After checking SQLAlchemy themselves,
http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#module-sqlalchemy.dialects.mysql.mysqldb
To use MySQL-Python on Python 3, they recommend a fork of it, mysqlclient,
https://github.com/PyMySQL/mysqlclient-python
It is available via pip with pip install mysqlclient, but there are almost certainly other steps you'll need to do to set it up initially. After that though, I was seeing the performance go back to what I was used to, which was about 5x faster than with Oracle's connector.
I've gotten oursql + SQLAlchemy 0.8.1 + Python 3.3 to work. Building off of LukeCarrier's port, I modified oursql.c to use the correct import levels, and it worked! Try this, and be sure to follow the readme:
https://github.com/clintron/py3k-oursql
You may also need to have the latest version of Cython.
Related
Problem
We're trying to connect to Cloud Spanner via SQLAlchemy version 1.3.23 and python-spanner-sqlalchemy. Using Poetry for dependency management, sqlalchemy-spanner has been added like so (this is how the project was set up):
sqlalchemy = "~1.3"
sqlalchemy-spanner = { git="https://github.com/cloudspannerecosystem/python-spanner-sqlalchemy.git", tag="v0.1.0" }
When create_engine is called with
create_engine("spanner:///projects/my-project/instances/my-instance/databases/my-db")
I get the following error
class 'sqlalchemy.exc.NoSuchModuleError'>", "NoSuchModuleError(\"Can't load plugin: sqlalchemy.dialects:spanner\")
Attempts
Registry
I've tried adding (as seen in the conftest.py file in the python-spanner-sqlalchemy test package)
from sqlalchemy.dialects import registry
registry.register("spanner", "google.cloud.sqlalchemy_spanner", "SpannerDialect")
before create_engine is called, which leads to the following error:
class 'ModuleNotFoundError'>", "ModuleNotFoundError(\"No module named 'google.cloud.sqlalchemy_spanner'\")
This makes me think that the plugin dialect has not been correctly added since, in line 49 of setup.py, the connection for the dialect is made:
entry_points={
"sqlalchemy.dialects": [
"spanner = google.cloud.sqlalchemy_spanner:SpannerDialect"
]
},
Installing via python setup.py install
In the README for the spanner project, it says to clone the repo and install via python setup.py install. I performed this step, but am unsure how to import this into my current project or make my project aware of this library. I've never manually installed python packages before so, if anyone can provide any help here, I'd appreciate it.
What I did try:
install the library as per above
try to add the dependency via poetry : poetry add sqlalchemy-spanner. Got Could not find a matching version of package sqlalchemy-spanner
try to locate the library via pip : pip install sqlalchemy-spanner== which usually lists available package versions.
I'm not sure that either of the last 2 bullets actually check a local installation of a package. Not even really sure what I'm talking about here.
Update
So I was able to install the local version of python-spanner-sqlalchemy by using pip install /path/to/project, which works, but still having the same issues with loading the dialect.
I added an import for SpannerDialect in the code (in the Registry section) above with from google.cloud.sqlalchemy_spanner import SpannerDialect. PyCharm auto-completed this for me which indicates to me that the package is successfully installed and available. But I receive the ModuleNotFoundError for google.cloud.sqlalchemy_spanner when running.
I ran python in my project root directory and, from the repl, imported SpannerDialect with no errors.
Solution
To clarify, the solution #larkee provided worked regarding the updated repository URL.
As a note, we recently moved the repo from cloudspannerecosystem/python-spanner-sqlalchemy to googleapis/python-spanner-sqlalchemy.
I clarified why that worked in the comments to their answer
I have not tested the answer from #neondot42, but I have seen this brought up as well, so take a look there if you're having the same issue.
sorry for the late response. I was recently struggling with a similar problem. Sqlalchemy was unable to load the spanner dialect. I tried uninstalling and installing different versions with pip but nothing seemed to work.
At the end what did the trick was specifying the "driver" part of the database URL as "spanner" too. So the final URL looked like this:
spanner+spanner:///projects/project-id/instances/instance-id/databases/database-id
I am not entirely sure of why this appeared to be the solution for me, as all the documentation I could find on Google's end just showed that you could use only the "spanner:///..." to connect. Also I am not familiar with the inner workings of sqlalchemy and how it should detect other installed dialects. However, I hope this solution can help someone else.
I was able to replicate the error you are seeing by installing sqlalchemy but not having sqlalchemy-spanner installed at all:
pip install sqlalchemy==1.3
pip uninstall sqlalchemy-spanner
>>> from sqlalchemy import create_engine
>>> engine = create_engine(<url>)
sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:spanner
After I install sqlalchemy-spanner I am able to use SQLAlchemy without issue:
pip install git+git://github.com/googleapis/python-spanner-sqlalchemy.git#v0.1.0
>>> from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String, inspect
>>> engine = create_engine(<url>)
>>> metadata = MetaData(bind=engine)
>>> table = Table(
... "TestTable",
... metadata,
... Column("user_id", Integer, primary_key=True),
... Column("user_name", String(16), nullable=False),
... )
>>> table.create()
>>> inspect(engine).get_table_names()
['TestTable']
>>>
Based on this, I think the issue is that sqlalchemy-spanner is not being installed. Unfortunately, I'm not familiar with Poetry for dependency management so I'm not sure exactly what is going wrong. As a note, we recently moved the repo from cloudspannerecosystem/python-spanner-sqlalchemy to googleapis/python-spanner-sqlalchemy. I am able to use either for the pip command but perhaps Poetry requires the newer one?
Within my Conda environment with Python 3.6.9, I've installed AllenNLP 9.2.0. I tried to install AllenNLP Server following the instruction from https://github.com/allenai/allennlp-server by running pip install --editable .
However, the installation procedure never finished as the compatibility checks with several modules, e.g. pip is looking at multiple versions of tqdm to determine which version is compatible with other requirements. This could take a while. Collecting tqdm>=4.19
Does anybody know what happens here? Should I add more restrictions to steup.py in AllenNLP server? However, there is any code included in such file.
Thanks a lot for your help.
I just tried it with AllenNLP 2.0.1 (the latest), and while it takes a long time, it does eventually resolve the packages.
That said, I would recommend two things:
Use Python 3.8 instead.
If it still doesn't work, specify a version of tqdm tightly in the requirements. My version automatically picked tqdm==4.56.2, just or reference.
I needed access a MySQL database via Jupyter Notebook, on which I run Python 3.6 (Anaconda install). It's a linear workflow, extracting data from the DB and manipulating it in Python/Pandas. No need for an ORM, a simple connector should do. However, the widely referenced MySQLdb package doesn't work with Python 3.x.
What are the alternatives?
The recommended installation modality for Jupyter on Ubuntu is Anaconda, so the appropriate package manager is conda. Installation via pip/pip3 or apt won't be accessible to the Notebook. conda makes it simple to get at least two good connectors:
pymysql works well and is easy to install:
sudo conda install pymysql
The 'official' connector:
sudo conda install mysql-connector-python
I tried pymysql first and it was fine but then switched to the second option due to the availability of extensive documentation.
If your objective is to import the data into a Pandas dataframe then use of the built-in pd.sql_read_table or pd.sql_read_query is convenient, as it labels the columns etc. It still requires installation of a connector, as discussed above.
An example with MySQL-connector-python, where you need to enter the database DETAILS:
import pandas as pd
import sqlalchemy
engine = sqlalchemy.create_engine('mysql+mysqlconnector://USER:PASSWORD#HOST/DB_NAME')
example_df = pd.read_sql_table("YOUR_TABLE_NAME", engine)
This earlier SO question suggests mysql.connector, which is what I am actually using. However, as noted in a comment to the accepted answer in the aforementioned question, this is not a standard python module.
My problem: whenever my ISP upgrades to a newer Python version, I have to notice the crashes and bug them to install mysql.connector for me.
I could explicitly call for an older version of Python 3.x. However, all other things being equal, I'd like to use the latest Python, and it seems like MySQL is just such a standard of standards that there really should be a standard Python 3 library for it. Is there? I'm not doing anything fancy with the database (or with python), so basic functionality is o.k.
Just use a venv and isolate yourself from ISP python upgrades:
pyvenv ./mysqlvenv
./mysqlvenv/bin/pip install mysql-connector-python --allow-external mysql-connector-python
install your other dependencies using ./mysqlvenv/bin/pip install foo and Bob's your uncle.
I am getting ImportError: No module named 'mysql' while I do the following...
>>> import mysql.connector
MySQL is installed and am on Python 3.5. I can't figure out. The above command is running fine in Python 2.7.
Unfortunately there is no mysql-connector available for python3.5.
So, you can use pymysql module which is a replacement for mysql-connector package
pip install pymysql
import pymysql
Connection = pymysql.connect(host='hostname', user='username', password='password',
db='database',charset='utf8mb4',cursorclass=pymysql.cursors.DictCursor )
ExecuteQuery(Connection)
Connection.close()
For me (OS X 10.11.4, Python 3.5.1), the mysql-connector-package installed itself (by default) into Python 2.7 that was also on my system. I copied the mysql package directory from:
/Library/Python/2.7/site-packages/mysql to /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/mysql. Everything works. According to http://dev.mysql.com/doc/connector-python/en/connector-python-versions.html the connector package supports Python 3.3+.
Your specific installation path for the module may be different, but you can easily check this from the REPL using the sys.path:
https://leemendelowitz.github.io/blog/how-does-python-find-packages.html
As we need to be connecting MySQl with Python, the Python command assumes that the connector is enabled. So we need to follow the steps below:
open MySQL Installer;
go to "Custom settings";
go to "Connectors";
choose the connector\Python and version based on the one you are using;
add it;
use import mysql.connector.
It should run, in my case it did!
From Can't run MySql Utilities:
MYSQL Utilities assumes that the MySQL Connector for Python has been installed. If you install it (http://dev.mysql.com/downloads/connector/python/), MySQL Utilities should run OK.
I think this link may help you to solve your problem.