I have a database with a lot of nonmanaged tables which I'm using for a django app. For testing I'm wanting to use the --keepdb option so that I don't have to repopulate these tables every time. I'm using MariaDB for my database. If I don't use the keepdb option everything works fine, the test database gets created and destroyed properly.
But when I try to run the test keeping the database:
$ python manage.py test --keepdb
I get the following error:
Using existing test database for alias 'default'...
Got an error creating the test database: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'CREATE DATABASE IF NOT EXISTS test_livedb ;\n SET sql_note' at line 2")
I assume that this is an issue with a different syntax between MariaDB and MySQL. Is there anyway to get the keepdb option to work with MariaDB?
thanks very much!
For what it's worth: This bug was introduced in Django 2.0.0 and fixed in Django 2.1.3 (https://code.djangoproject.com/ticket/29827)
Two things - check out Factory Boy (for creating test data) and I would suggest checking out Pytest as well. With non-managed tables, the issue I think you'll run into is that (at least in my experience) django won't create them in the test environment and you end up running into issues because there is no migration file to create those tables (since they're unmanaged). Django runs the migration files when creating the test environment.
With Pytest you can run with a --nomigrations flag which builds your test database directly off the models (thus creating the tables you need for your unmanaged models).
If you combine Pytest and Factory Boy you should be able to come up with the ability to setup your test data so it works as expected, is repeatable and testable without issue.
I actually approach it like this (slightly hacky, but it works with our complex setup):
On my model:
class Meta(object):
db_table = 'my_custom_table'
managed = getattr(settings, 'UNDER_TEST', False)
I create the UNDER_TEST variable in settings.py like this:
# Create global variable that will tell if our application is under test
UNDER_TEST = (len(sys.argv) > 1 and sys.argv[1] == 'test')
That way - when the application is UNDER_TEST the model is marked as managed (and Pytest will create the appropriate DB table). Then FactoryBoy handles putting all my test data into that table (either in setUp of the test or elsewhere) so I can test against it.
That's my suggestion - others might have something a little more clear or cleaner.
Related
I'm getting the following error message when trying to reflect any of my SQL views:
sqlalchemy/dialects/mysql/reflection.py", line 306, in _describe_to_create
buffer.append(" ".join(line))
TypeError: sequence item 2: expected str instance, bytes found
I have tried using both the autoload_with and autoload=True options in my select query constructor to no avail.
I have the appropriate permissions on my view. My query is pretty simple:
company_country = Table('company_country', metadata, autoload_with=engine)
query = select(company_country.c.country)
return query
I've tried the inspect utility and it does not list my SQL view, nor does the reflecting all tables described below the views section on this page: https://docs.sqlalchemy.org/en/14/core/reflection.html#reflecting-views
I'm using version SQLAlchemy->1.4.32, Python 3.x and mySQL 8.0.28 on Mac if that's any help
I should add that I can query my SQL views using the text() constructor but it would be far more preferable to use select() if possible.
Any tips appreciated
I was using the mysql-connector client for interop with other code, but after switching to the mysqlclient, I was able to reflect the views.
https://docs.sqlalchemy.org/en/14/dialects/mysql.html#module-sqlalchemy.dialects.mysql.mysqldb
In PyCharm I created a MySQL schema using pymysql on my computer. Now I want to use Peewee to create tables and write the SQL queries. However, I always receive an error message (see below) when trying to connect to the DB.
The user has sufficient rights to create tables in the DB schema as it works flawlessly with pymysql (creating tables as well as the schema works fine).
I looked at similar questions on Stackoverflow and couldn't find a similar problem. Moreover, this problem wasn't experienced in any of the tutorials I looked at, so I'm not entirely sure what could be the culprit causing the error. Below is a minimal working example.
from peewee import*
import peewee
user = 'root'
password = 'root'
db_name = 'peewee_demo'
# The schema with the name 'peewee_demo' exists
db = MySQLDatabase(db_name, user=user, passwd=password)
class Book(peewee.Model):
author = peewee.CharField()
title = peewee.TextField()
class Meta:
database = db
db.connect() # Code fails here
Book.create_table()
book = Book(author="me", title='Peewee is cool')
book.save()
for book in Book.filter(author="me"):
print(book.title)
I would expect the above code to connect to MySQL and then create a new table in the schema "peewee_demo". But instead, the code throws an error message when trying to connect to the DB:
/usr/bin/python3.6: Relink '/lib/x86_64-linux-gnu/libsystemd.so.0' with /lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol clock_gettime'
/usr/bin/python3.6: Relink /lib/x86_64-linux-gnu/libudev.so.1' with /lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'
Do you have any ideas how to fix this issue?
Thanks in advance
As #coleifer pointed out in his comment, the error was probably related to a shared library issue in Python. After setting up a new virtual environment and installing all required packages, everything runs perfectly fine.
I just added the answer to be able to close the question. If #coleifer converts his comment into an answer, I'll delete mine and accept his.
I am in the process of migrating databases from sqlite to mysql. Now that I've migrated the data to mysql, I'm not able to use my sqlalchemy code (in Python3) to access it in the new mysql db. I was under the impression that sqlalchemy syntax was database agnostic (i.e. the same syntax would work for accessing sqlite and mysql), but this appears not to be the case. So my question is: Is it absolutely required to use a DBAPI in addition to Sqlalchemy to read the data? Do I have to edit all of my sqlalchemy code to now read mysql?
The documentation says: The MySQL dialect uses mysql-python as the default DBAPI. There are many MySQL DBAPIs available, including MySQL-connector-python and OurSQL, which I think means that I DO need a DBAPI.
My old code with sqlite successfully worked like this with sqlite:
engine = create_engine('sqlite:///pmids_info.db')
def connection():
conn = engine.connect()
return conn
def load_tables():
metadata = MetaData(bind=engine) #init metadata. will be empty
metadata.reflect(engine) #retrieve db info for metadata (tables, columns, types)
inputPapers = Table('inputPapers', metadata)
return inputPapers
inputPapers = load_tables()
def db_inputPapers_retrieval(user_input):
result = engine.execute("select title, author, journal, pubdate, url from inputPapers where pmid = :0", [user_input])
for row in result:
title = row['title']
author = row['author']
journal = row['journal']
pubdate = row['pubdate']
url = row['url']
apa = str(author+' ('+pubdate+'). '+title+'. '+journal+'. Retrieved from '+url)
return apa
This worked fine and dandy. So then I tried to update it to work with the mysql db like this:
engine = create_engine('mysql://snarkshark#localhost/pmids_info')
At first when I tried to run my sample code like this, it complained because I didn't have MySqlDB. Some googling around informed me that MySqlDB does NOT work for Python 3. So then I tried pip installing pymysql and changing my engine statement to
engine = create_engine('mysql+pymysql://snarkshark#localhost/pmids_info')
which also ends up giving me various syntax errors when I try to adjust things.
So what I want to know, is if there is any way I can get my current syntax to work with mysql? Since the syntax is from sqlalchemy, I thought it would work perfectly for the exact same data in mysql that was previously in sqlite. Will I have to go through and update ALL of my db functions to use the syntax of the DBAPI?
This will sound like a dumb answer, but you'll need to change all the places where you're using database-specific behavior. SQLAlchemy does not guarantee that anything you do with it is portable across all backends. It leaks some abstractions on purpose to allow you to do things that are only available on certain backends. What you're doing is like using Python because it's cross-platform, then doing a bunch of os.fork()s everywhere, and then being surprised that it doesn't work on Windows.
For your specific case, at a minimum, you need to wrap all your raw SQL in text() so that you're not affected by the supported paramstyle of the DBAPI. However, there are still subtle differences between different dialects of SQL, so you'll need to use the SQLAlchemy SQL expression language instead of raw SQL if you want portability. After all that, you'll still need to be careful not to use backend-specific features in the SQL expression language.
I have SQLite and MySQL installed on my local and development machine respectively. Following is working fine on my local machine(with SQLite):
select_single = {'date': "strftime('%%Y-%%m-%%d',projectName_Modelname.created)"}
queryset.extra(select=select_single)
But since strftime doesn't work with MySQL(link), I tried using DATE_FORMAT() as suggested in given link and other places too.
Though now when I execute below:
select_single = {'date': "DATE_FORMAT(projectName_Modelname.created, '%%Y-%%m-%%d')"}
queryset.extra(select=select_single)
Following error comes:
DatabaseError: (1054, "Unknown column 'projectName_Modelname.created' in 'field list'")
where 'created' is Datetime field in Django model 'Modelname' of app 'projectName'
To debug when I replace projectName_Modelname.created with NOW() no error comes. I have also tried just Modelname.created instead of projectName_Modelname.created though with no benefit?
Note: I am using Django1.5.5
I think it should be something like:
date_raw_query = {'date': "date_format(created, '%%Y-%%m-%%d')"}
and then try
queryset.extra(select=date_raw_query)
Hope that works in your setup. I have tried this on Django 1.7 and MySQL and seems to be working.
Also remember that if SQL errors start coming up, you can always do a print queryset.extra(select=date_raw_query).query to see what might be going wrong.
And when it comes to writing compatible code between SQLite and MySQL like this one, writing a custom MySQL function has been suggested here
But I would suggest otherwise. It's better to have a similar dev environment with MySQL setup in local and also, upgrade Django as soon as possible. :P
I'm experiencing a strange bug with the way Django Test framework operates.
When using SQLite Database Backend, all of the tests crash with the following error:
File "[]/core/tests/test_admin.py", line 91, in setUpSomething
content_type = ContentType.objects.get(app_label='core', model='SomeModel')
File "[]/lib/python2.7/site-packages/django/db/models/manager.py", line 151, in get
return self.get_queryset().get(*args, **kwargs)
File "[]/lib/python2.7/site-packages/django/db/models/query.py", line 310, in get
self.model._meta.object_name)
DoesNotExist: ContentType matching query does not exist.
However, the same code executes well under MySQL backend.
Clearly Django should make these functions agnostic of the backend used?
I had the same problem and I have no idea if my solution will be helpful, but it solved my issue, so here goes.
In my application code, I was attempting to query ContentType instances in the following way:
email = ContentType.objects.get(app_label="users", model="EmailAddress")
This worked fine with our actual MySQL database, but failed in test under a SQLite test database. However, if I switched the model definition to lowercase, it worked in both places:
email = ContentType.objects.get(app_label="users", model="emailaddress")
My guess was that this may have to do with the default collation in MySQL of case-insensitivity, so the first query should not have worked if I were comparing case-sensitively.
Indeed, when I looked at my database, all the model labels in the django_content_type table were lowercase and SQLite cares (by default) about case, so my queries in my tests were legitimately failing.