migrate in django to legacy database gives error - mysql

Background info:
I am quite beginner with django, better equipped with python and quite beginner in mysql (and thus mariaDB) as well.
I work in Windows 7 environment with the following versions:
django 1.9.1
mariaDB 10.0.20
Python 3.4.4
mysql-connector\Python 2.1.3 for Python 3.4
Goal:
I am trying to build mariaDB database and web based user interface for it with django. I have built the database beforehand in mysql and am now trying to create the django project using this database as legacy database.
Problem:
I have retrieved the models to my django project as follows:
python manage.py inspectdb > models.py
and it worked fine. Checked the models.py, as well, and they seemed as they should.
Now, if I try to migrate to create the tables based on django to my legacy db as follows:
python manage.py migrate
I keep getting plenty of nested errors last one being:
django.db.migrations.exceptions.MigrationsSchemaMissing: Unable to create the django_migrations table (Table 'mydb'.'django_migrations' already exists. Please DISCARD the tablespace before IMPORT.)
I figured the table was left in my database folder from previous attempt, but removing it does not help. Even if I check from the database command prompt that the only tables left are native to my database, this same error message pops up still. At some point I wondered if there is some issue with the connector I am using as it is not the recommended one (mysqlclient).
Somewhere, I found that the migration would not even be required. This would obviously save me, but I have no idea, if that is a solution I can go for. Ideally the database would be managed later online and users with admin accounts could add entries there. Otherwise, however the database should be just returning the data.
EDIT: Fixed the error message the python manage.py inspectdb > models.py line
EDIT 29.1.
Running the migration as:
python manage.py migrate --fake
results in the same error.
The issue persists even after removing the django project and starting it fresh.
EDIT v2 29.1.
There is only one of the required table files for the django_migrations table within the database folder. There should be django_migrations.ibd AND django_migrations.frm. However, after running the migration, there is only one file: django_migrations.ibd.
There was something similar going on here, with different file
mysqldump problems with restore error: 'Please DISCARD the tablespace before IMPORT'. It is well explained in the chosen answer.
I think this is related to my issue, but cannot figure out what could help and what to do with this.
EDIT v3 29.1.
And finally I noticed the file giving the django error. It is django's migration recorder at C:\Python34\lib\site-packages\django\db\migrations\recorder.py on line 59.
def ensure_schema(self):
"""
Ensures the table exists and has the correct schema.
"""
# If the table's there, that's fine - we've never changed its schema
# in the codebase.
if self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()):
return
# Make the table
try:
with self.connection.schema_editor() as editor:
editor.create_model(self.Migration)
except DatabaseError as exc:
raise MigrationSchemaMissing("Unable to create the django_migrations table (%s)" % exc)
Where the last line raising the error is the line 59. Funnily, the function should work, if the table exists. Only thing I can come up with (as in my previous edit), is that indeed for some reason the frm-file is not created during the migration process.
Why that happens is way beyond me.

Related

Flask migration error when using flask-migrate

I can't seem to get Flask to migrate my models. I'm following along with the Flask Mega-Tutorial series. At the database setup part of the tutorial, I'm just trying to substitute MySQL for SQLite that is used in the tutorial.
I followed SQLAlchemy's instructions for connecting to a MySQL database, and I've included mysqlclient in my pipfile.
But when I run:
flask db init
flask db migrate
I get the following:
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
ERROR [root] Error: Can't locate revision identified by '7bb962b87f19'
I've tried deleting the migrations folder, and I've deleted the environment and recreated to see if that would fix anything. But it keeps saying it can't locate revision. I thought maybe it did something with the database, I've read other solutions saying you have to flush the database. But, there aren't any tables created or any schemas in the database.
Other info:
DATABASE_URL="mysql://root:mypassword#localhost/flask_tutorial"
SQLite database runs fine and makes the migrations. So it must be something I've done with the mysql setup.
Any suggestions or ideas that may lead me in the right direction would be greatly appreciated.
The session was storing incorrect environmental variables exported. Had to completely restart my machine for the settings to reset. I even tried unset for the variables, which didn't remove the variables. Once Flask was able to pick up the correct variable settings for my configuration everything worked as planned.
Please make sure you have used this "SQLALCHEMY_TRACK_MODIFICATIONS= True"
Did you install MySqldb lib. to connect with your mysql db ?
I had the same issue. I had deleted the migrations folder but hadn't done the same for the app.db file. So, every time I tried to create any db migrations, it failed because it was using the old (and still existing) app.db file.
This is how my database was configured:
class Config(object):
# Database configuration
SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL') or \
'sqlite:///' + os.path.join(basedir, 'app.db')
SQLALCHEMY_TRACK_MODIFICATIONS = False
Deleting the app.db file solved the error. Subsequent db migrations and upgrades worked well.

Django 1.10 raises error when installing fixtures: no such column: REFERRED.id

I created a fixture with this command:
python manage.py dumpdata --natural-primary --natural-foreign -e admin -e contenttypes -e auth.Permission --format=json --indent=2 --all > gc_core/fixtures/initial_data.json
Then I tried to test it on my other machine. So I deleted my database and run migrations there:
./manage.py migrate
Then I tried to populate my database with that fixture using loaddata:
./manage.py loaddata gc_core/fixtures/initial_data.json
Then Django raises this error:
django.db.utils.OperationalError: Problem installing fixtures: no such column: REFERRED.id
I have no idea what this means. Should I have a column somewhere named REFERRED.id?
There is no such column in my fixture nor in the database and I didn't defined such model in any of my apps.
So what is REFERRED.id and how do I fix it?
I'm using Django 1.10, Python 3.5.2, SQLite. The fixture I'm trying to load is in JSON format.
Thank you in advance.
I stumbled upon this question when I encountered the same issue.
Here is a possible answer that is likely just a theory but I will explain why.
Django is having trouble loading relationships with primary keys that are not auto indexes on SQLite databases.
I am encountering this issue because the test database strategy we employ is by using an SQLite database instead of the production like Postgres database. I have been breaking my head over the exact same error all day. When I switched to Postgres for the test database as well, it ended up not being an issue. There might be a bigger problem at large here. Hopefully someone else can provide a more clear answer.
I am using the same version of Django and Python 2.7
If possible, could you try moving the data into a Postgres equivalent and load it. If that does work, it means your data is not wrong, and the relationship SQLite and Django is having issues.

Django 1.8 migrations "Table already exists"

I've been working on making an auto-deployment system to make updating my django site easier, and it mostly works, but right now my database changes are screwy. I have the auto deployment run "makemigrations" "migrate" then "syncdb"
The trouble I have is when I run migrate, it tells me that my table already exists and can't be created. Previous answers have said to run "migrate --fake", but if I do this, it appears to think everything is up to date. The issue then is that I run "makemigrations" or "migrate" and it says no changes detected (even though there is a missing column from my database). I tried to run "sqlall" to figure out what it thinks the database should be, and it tells me that I have pending migrations. So I tried running "migrate" and it said no migrations exist.
How do I manage this? And in the future, what should I do for database migrations to troubleshoot/fix these problems?
Thanks
I fixed the problem. The issue stemmed from a misunderstanding of how Django migrations worked. Also, as per the comments, syncdb is no longer a command in manage.py.
The original issue was that I had been using a different database for deployment and then I switched and my migrations were out of sync from my development database, so I had to clear out both databases and migrations and start over with my data. Then I run makemigrations on my local machine and migrate there (for my dev DB). Then I upload to the server and have the fab script run migrate on the server production DB without calling makemigrations. This seems to have fixed the issue.

How do I migrate data from production db to development db (Rails 4)?

I know this one sounds kind of backwards! I've been working on a cloud-based case-management application, and was working on a support-ticketing feature. We have our development database (MySQL) that has all the same data as our production database (It's a very large app). Development is basically a "sandbox" environment, hence why the development db has all the same as the production db. This morning, I ran into a problem in my local development server for:
Migrations are pending; run 'bin/rake db:migrate RAILS_ENV=development' to resolve this issue.
Alright, did that, even though it didn't make any sense. It err'd out, because it was trying to create tables that already existed (I had already ran my migrations, the features were completed a day ago! And everything worked great).
The only thing that I did the moment before I got this message in my development server was uncomment an entirely-commented-out file to try and fix problems with TinyMCE (/config/tinymce.yml, text editor), followed by restarting my rails dev server. All this file was, was a bunch of defaults and plugins. When I saw the problem, I first commented them all back out. Still the same error. Proceeded with rake db: tasks.
Started looking for answers.. kept getting errors. Made a noob mistake, and ultimately did a
rake db:migrate:reset
which I knew would rebuild the schema at the cost of hosing the db. This failed.. But we do have the db in more than one place (since it's the same as production.) Also, just to try and bring all the tables back:
rake db:schema:load
....failed on one of the tables anyway with
ActiveRecord::StatementInvalid: Mysql2::Error: Specified key was too long; max key length is 767 bytes: CREATE UNIQUE INDEX `login` USING btree ON `users` (`login`, `intranet_id`)
..and all the tables are empty (as expected) that were successfully created. I'm by no means a database pro (I haven't dev'd very long). Am wondering, is there a way to copy the entire production database into the devel database? Tables, data, and all. Or any other suggestions? Really lost on this one.
You can use mysql_dump.
If you connect to the database everything is there - you just have to do a dump of the full database and then do the dump again into the production tables.
As you clearly see you can output from the database with ">" and you can import with the "<".

mysql alter table stuck in preparing state

I recently tried to change a column in my database with a south migration for my django project from a CharField to a IntegerField
I did a python manage.py schemamigration appname --auto
i then applied the migration and it just hung. I stupidly ran this right on my production server before trying out locally. Once I tried locally I got this error in my traceback
File "/Library/Python/2.7/site-packages/MySQL_python-1.2.3-py2.7-macosx-10.7-intel.egg/MySQLdb/cursors.py", line 92, in _warning_check
_mysql_exceptions.Warning: Data truncated for column 'pickup_id' at row 1
When I go into mysql workbench and just try to revert the changes back to a CharField it just hangs in the "preparing" state and eventually times out. My site just hangs wherever this table is used and I cant preform any queries on this particular table.
I'm open to any suggestion. Thank you
What ended up working was applying the --fake parameters to my migrate call
./manage migrate appname --fake