I'm getting the following error message when trying to reflect any of my SQL views:
sqlalchemy/dialects/mysql/reflection.py", line 306, in _describe_to_create
buffer.append(" ".join(line))
TypeError: sequence item 2: expected str instance, bytes found
I have tried using both the autoload_with and autoload=True options in my select query constructor to no avail.
I have the appropriate permissions on my view. My query is pretty simple:
company_country = Table('company_country', metadata, autoload_with=engine)
query = select(company_country.c.country)
return query
I've tried the inspect utility and it does not list my SQL view, nor does the reflecting all tables described below the views section on this page: https://docs.sqlalchemy.org/en/14/core/reflection.html#reflecting-views
I'm using version SQLAlchemy->1.4.32, Python 3.x and mySQL 8.0.28 on Mac if that's any help
I should add that I can query my SQL views using the text() constructor but it would be far more preferable to use select() if possible.
Any tips appreciated
I was using the mysql-connector client for interop with other code, but after switching to the mysqlclient, I was able to reflect the views.
https://docs.sqlalchemy.org/en/14/dialects/mysql.html#module-sqlalchemy.dialects.mysql.mysqldb
Related
I have a database with a lot of nonmanaged tables which I'm using for a django app. For testing I'm wanting to use the --keepdb option so that I don't have to repopulate these tables every time. I'm using MariaDB for my database. If I don't use the keepdb option everything works fine, the test database gets created and destroyed properly.
But when I try to run the test keeping the database:
$ python manage.py test --keepdb
I get the following error:
Using existing test database for alias 'default'...
Got an error creating the test database: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'CREATE DATABASE IF NOT EXISTS test_livedb ;\n SET sql_note' at line 2")
I assume that this is an issue with a different syntax between MariaDB and MySQL. Is there anyway to get the keepdb option to work with MariaDB?
thanks very much!
For what it's worth: This bug was introduced in Django 2.0.0 and fixed in Django 2.1.3 (https://code.djangoproject.com/ticket/29827)
Two things - check out Factory Boy (for creating test data) and I would suggest checking out Pytest as well. With non-managed tables, the issue I think you'll run into is that (at least in my experience) django won't create them in the test environment and you end up running into issues because there is no migration file to create those tables (since they're unmanaged). Django runs the migration files when creating the test environment.
With Pytest you can run with a --nomigrations flag which builds your test database directly off the models (thus creating the tables you need for your unmanaged models).
If you combine Pytest and Factory Boy you should be able to come up with the ability to setup your test data so it works as expected, is repeatable and testable without issue.
I actually approach it like this (slightly hacky, but it works with our complex setup):
On my model:
class Meta(object):
db_table = 'my_custom_table'
managed = getattr(settings, 'UNDER_TEST', False)
I create the UNDER_TEST variable in settings.py like this:
# Create global variable that will tell if our application is under test
UNDER_TEST = (len(sys.argv) > 1 and sys.argv[1] == 'test')
That way - when the application is UNDER_TEST the model is marked as managed (and Pytest will create the appropriate DB table). Then FactoryBoy handles putting all my test data into that table (either in setUp of the test or elsewhere) so I can test against it.
That's my suggestion - others might have something a little more clear or cleaner.
I am in the process of migrating databases from sqlite to mysql. Now that I've migrated the data to mysql, I'm not able to use my sqlalchemy code (in Python3) to access it in the new mysql db. I was under the impression that sqlalchemy syntax was database agnostic (i.e. the same syntax would work for accessing sqlite and mysql), but this appears not to be the case. So my question is: Is it absolutely required to use a DBAPI in addition to Sqlalchemy to read the data? Do I have to edit all of my sqlalchemy code to now read mysql?
The documentation says: The MySQL dialect uses mysql-python as the default DBAPI. There are many MySQL DBAPIs available, including MySQL-connector-python and OurSQL, which I think means that I DO need a DBAPI.
My old code with sqlite successfully worked like this with sqlite:
engine = create_engine('sqlite:///pmids_info.db')
def connection():
conn = engine.connect()
return conn
def load_tables():
metadata = MetaData(bind=engine) #init metadata. will be empty
metadata.reflect(engine) #retrieve db info for metadata (tables, columns, types)
inputPapers = Table('inputPapers', metadata)
return inputPapers
inputPapers = load_tables()
def db_inputPapers_retrieval(user_input):
result = engine.execute("select title, author, journal, pubdate, url from inputPapers where pmid = :0", [user_input])
for row in result:
title = row['title']
author = row['author']
journal = row['journal']
pubdate = row['pubdate']
url = row['url']
apa = str(author+' ('+pubdate+'). '+title+'. '+journal+'. Retrieved from '+url)
return apa
This worked fine and dandy. So then I tried to update it to work with the mysql db like this:
engine = create_engine('mysql://snarkshark#localhost/pmids_info')
At first when I tried to run my sample code like this, it complained because I didn't have MySqlDB. Some googling around informed me that MySqlDB does NOT work for Python 3. So then I tried pip installing pymysql and changing my engine statement to
engine = create_engine('mysql+pymysql://snarkshark#localhost/pmids_info')
which also ends up giving me various syntax errors when I try to adjust things.
So what I want to know, is if there is any way I can get my current syntax to work with mysql? Since the syntax is from sqlalchemy, I thought it would work perfectly for the exact same data in mysql that was previously in sqlite. Will I have to go through and update ALL of my db functions to use the syntax of the DBAPI?
This will sound like a dumb answer, but you'll need to change all the places where you're using database-specific behavior. SQLAlchemy does not guarantee that anything you do with it is portable across all backends. It leaks some abstractions on purpose to allow you to do things that are only available on certain backends. What you're doing is like using Python because it's cross-platform, then doing a bunch of os.fork()s everywhere, and then being surprised that it doesn't work on Windows.
For your specific case, at a minimum, you need to wrap all your raw SQL in text() so that you're not affected by the supported paramstyle of the DBAPI. However, there are still subtle differences between different dialects of SQL, so you'll need to use the SQLAlchemy SQL expression language instead of raw SQL if you want portability. After all that, you'll still need to be careful not to use backend-specific features in the SQL expression language.
I have SQLite and MySQL installed on my local and development machine respectively. Following is working fine on my local machine(with SQLite):
select_single = {'date': "strftime('%%Y-%%m-%%d',projectName_Modelname.created)"}
queryset.extra(select=select_single)
But since strftime doesn't work with MySQL(link), I tried using DATE_FORMAT() as suggested in given link and other places too.
Though now when I execute below:
select_single = {'date': "DATE_FORMAT(projectName_Modelname.created, '%%Y-%%m-%%d')"}
queryset.extra(select=select_single)
Following error comes:
DatabaseError: (1054, "Unknown column 'projectName_Modelname.created' in 'field list'")
where 'created' is Datetime field in Django model 'Modelname' of app 'projectName'
To debug when I replace projectName_Modelname.created with NOW() no error comes. I have also tried just Modelname.created instead of projectName_Modelname.created though with no benefit?
Note: I am using Django1.5.5
I think it should be something like:
date_raw_query = {'date': "date_format(created, '%%Y-%%m-%%d')"}
and then try
queryset.extra(select=date_raw_query)
Hope that works in your setup. I have tried this on Django 1.7 and MySQL and seems to be working.
Also remember that if SQL errors start coming up, you can always do a print queryset.extra(select=date_raw_query).query to see what might be going wrong.
And when it comes to writing compatible code between SQLite and MySQL like this one, writing a custom MySQL function has been suggested here
But I would suggest otherwise. It's better to have a similar dev environment with MySQL setup in local and also, upgrade Django as soon as possible. :P
I am constructing a url for a CMS data query. I get the following to work just fine:
https://data.cms.gov/resource/din4-7td8.json?$$app_token=REDACTED&$where=(starts_with(nppes_provider_zip,'63703') OR starts_with(nppes_provider_zip,'63701')) AND (hcpcs_code='31623' OR hcpcs_code='31622')
When I try to substitute calling multiple hcpcs_code values, I get a query.compiler.malformed error. The following generates the error:
https://data.cms.gov/resource/din4-7td8.json?$$app_token=REDACTED&$where=(starts_with(nppes_provider_zip,'63703') OR starts_with(nppes_provider_zip,'63701') AND hcpcs_code in('31622','31623'))
Is it possible that I am using the in(...) function incorrectly?
I think I may have caught you on IRC after you asked this question, but I'll answer here too.
It looks like you're using an old version of that API endpoint which doesn't support the IN(...) function. If you migrate to the new version of that dataset API you'll be able to issue your query:
http://dev.socrata.com/foundry/#/data.cms.gov/5fnr-qp4c
https://data.cms.gov/resource/5fnr-qp4c.json?$where=(starts_with(nppes_provider_zip,'63703') OR starts_with(nppes_provider_zip,'63701') AND hcpcs_code in('31622','31623'))
I've inherited a Pylons app that uses SQLAlchemy. I know nothing about SQLAlchemy and very little about Pylons :)
I need to run some raw SQL from within the app. The SQLAlchemy currently seems to be working in the following way (example code):
import myapp.model as model
model.Session.query(model.KeyValue) # existing code
.join(model.Key)
.filter(model.Key.name == name)
).count() == 0, name
How do I get it to run raw SQL? I see that I need an execute() statement, but how exactly do I run it? The following both fail:
model.Session.execute('create table hello_world;')
model.Connection.execute("""
create table hello_world;
""")
What's the magic invocation? There's no reference to a Connection object in the existing code, and I'm not sure how to create one.
You can obtain connection that is used by Session by using its connection method:
connection = model.Session.connection()
Then you can issue your query:
connection.execute('create table hello_world;')
Note that in Pylons model.Session is not a sqlalchemy.orm.session.Session class. It's an instance of sqlalchemy.orm.scoping.ScopedSession. That's how it's created in model.meta module:
Session = scoped_session(sessionmaker())
My first impulse is to recommend trying the execute() method of an instance of Connection, instead of the execute() method of the class itself as your example code suggests that you're doing.
Are you working off of the Pylons Book examples ?