South - migrating django application from sqlite to mysql - mysql

I have a django application and until now I used sqLite as database backend. Now when it's quite close to production I thought of moving it all to mySQL which will be used on the box.
I reconfigured my settings to a mysql database and run
manage.py syncdb --migrate
it started creating tables and all but failed in the first (out of 40) migration with an old error (can't insert blob without key length and all that).
I first thought of manually fixing the migration files but quickly realized there is too much manual work.
So I thought ok I'll run manage.py migrate core 0040 (the last migration and that will do the trick) but it still attempts to run the initial one as well:
File "D:\~Sasha\Portman\core\migrations\0001_initial.py", line 23, in forwards
('name', self.gf('django.db.models.fields.TextField')(unique=True)),
... error message
Is there a way to migrate my models somehow without manually fixing the initial migration file and all the other magic?

First, you can't pick and choose migrations to run. migrate core 0040 means run all migrations up to 0040. In other words, it wouldn't run 0041, but it would run 0001-0040.
Now, it's side-stepping your question a bit, but if you have not moved this project to production yet, you don't actually need all those migrations. Assuming they're all schemamigrations you can rollback to zero with:
python manage.py migrate core zero
Then, delete them all (including 0001_initial.py) and simply run again:
python manage.py schemamigration --initial core
To regenerate the initial migration. It will base it off the current state of your models, negating the need for 40 migrations.
It's always a good idea to compress your migrations like this before you move new code to production. Since this is the first launch, you can remove them all and start from scratch, but in future iterations, if you generate 5 migrations during the course of development, before you commit, rollback to before the first of those, then delete those 5 and then generate a new schemamigration. The result is just one migration with all the changes of those 5. Then, you can commit that and migrate in production.
It may not solve your problem completely here, but it will definitely make things simpler to debug.

Very occasionally, I've seen problems with migrations when deploying a project on a mysql backend.
Since you are deploying a fresh copy, you do have a couple options for sidestepping the need to run all of your mightations:
First, if you want to maintain your migration history as-is, you can force syncdb to create all of your tables regardless of migrations, and then run fake migrations to get your new database up to date. That would look like:
python manage.py syncdb --all
python manage.py migrate --fake
Alternatively, if you don't care about your historical migrations, you can clear the old migrations out (just delete them) and then create a new initial migration, like:
python manage.py schemamigration --initial core
Just be sure that you also clear out the south_migrationhistory table in your dev database so that everything stays in sync between dev and prod

Related

migrate in django to legacy database gives error

Background info:
I am quite beginner with django, better equipped with python and quite beginner in mysql (and thus mariaDB) as well.
I work in Windows 7 environment with the following versions:
django 1.9.1
mariaDB 10.0.20
Python 3.4.4
mysql-connector\Python 2.1.3 for Python 3.4
Goal:
I am trying to build mariaDB database and web based user interface for it with django. I have built the database beforehand in mysql and am now trying to create the django project using this database as legacy database.
Problem:
I have retrieved the models to my django project as follows:
python manage.py inspectdb > models.py
and it worked fine. Checked the models.py, as well, and they seemed as they should.
Now, if I try to migrate to create the tables based on django to my legacy db as follows:
python manage.py migrate
I keep getting plenty of nested errors last one being:
django.db.migrations.exceptions.MigrationsSchemaMissing: Unable to create the django_migrations table (Table 'mydb'.'django_migrations' already exists. Please DISCARD the tablespace before IMPORT.)
I figured the table was left in my database folder from previous attempt, but removing it does not help. Even if I check from the database command prompt that the only tables left are native to my database, this same error message pops up still. At some point I wondered if there is some issue with the connector I am using as it is not the recommended one (mysqlclient).
Somewhere, I found that the migration would not even be required. This would obviously save me, but I have no idea, if that is a solution I can go for. Ideally the database would be managed later online and users with admin accounts could add entries there. Otherwise, however the database should be just returning the data.
EDIT: Fixed the error message the python manage.py inspectdb > models.py line
EDIT 29.1.
Running the migration as:
python manage.py migrate --fake
results in the same error.
The issue persists even after removing the django project and starting it fresh.
EDIT v2 29.1.
There is only one of the required table files for the django_migrations table within the database folder. There should be django_migrations.ibd AND django_migrations.frm. However, after running the migration, there is only one file: django_migrations.ibd.
There was something similar going on here, with different file
mysqldump problems with restore error: 'Please DISCARD the tablespace before IMPORT'. It is well explained in the chosen answer.
I think this is related to my issue, but cannot figure out what could help and what to do with this.
EDIT v3 29.1.
And finally I noticed the file giving the django error. It is django's migration recorder at C:\Python34\lib\site-packages\django\db\migrations\recorder.py on line 59.
def ensure_schema(self):
"""
Ensures the table exists and has the correct schema.
"""
# If the table's there, that's fine - we've never changed its schema
# in the codebase.
if self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()):
return
# Make the table
try:
with self.connection.schema_editor() as editor:
editor.create_model(self.Migration)
except DatabaseError as exc:
raise MigrationSchemaMissing("Unable to create the django_migrations table (%s)" % exc)
Where the last line raising the error is the line 59. Funnily, the function should work, if the table exists. Only thing I can come up with (as in my previous edit), is that indeed for some reason the frm-file is not created during the migration process.
Why that happens is way beyond me.

Django 1.8 migrations "Table already exists"

I've been working on making an auto-deployment system to make updating my django site easier, and it mostly works, but right now my database changes are screwy. I have the auto deployment run "makemigrations" "migrate" then "syncdb"
The trouble I have is when I run migrate, it tells me that my table already exists and can't be created. Previous answers have said to run "migrate --fake", but if I do this, it appears to think everything is up to date. The issue then is that I run "makemigrations" or "migrate" and it says no changes detected (even though there is a missing column from my database). I tried to run "sqlall" to figure out what it thinks the database should be, and it tells me that I have pending migrations. So I tried running "migrate" and it said no migrations exist.
How do I manage this? And in the future, what should I do for database migrations to troubleshoot/fix these problems?
Thanks
I fixed the problem. The issue stemmed from a misunderstanding of how Django migrations worked. Also, as per the comments, syncdb is no longer a command in manage.py.
The original issue was that I had been using a different database for deployment and then I switched and my migrations were out of sync from my development database, so I had to clear out both databases and migrations and start over with my data. Then I run makemigrations on my local machine and migrate there (for my dev DB). Then I upload to the server and have the fab script run migrate on the server production DB without calling makemigrations. This seems to have fixed the issue.

SyncDB after installing new app having already installed South - don't want to mess anything up

I'm using South to manage my (MySQL) database tables for a Django 1.4 project, its working great.
This is a bit of a newbie question, but I'm now adding sorl.thumbnail (http://sorl-thumbnail.readthedocs.org/en/latest/installation.html#installation) to the list of installed apps in the settings file.
The instructions say that I now have to use syncdb if I'm using a "cached database key value store".
Is it OK to go ahead and use syncdb? I'm not quite sure if my MySQL+South installation counts as one. Will this mess anything up?
If i'm not wrong, when you start using south you should never use syncdb again. Instead, you should use schemamigration or other south's specific commands.
Here you can find what you were looking for. I quote here the phrase that should clear your mind:
The main use of schemamigration is when you’ve just finished your shiny new models.py and want to load up your database. In vanilla Django, you’d just run syncdb - however, with migrations, you’ll need a migration to create the tables.
In this scenario, you just run:
./manage.py schemamigration myapp --initial
That will write one big migration to create all the tables for the
models in your app; just run ./manage.py migrate to get it in and
you’re done in only one more step than syncdb!
Hope it helps!
syncdb does not interfere with South, in fact, in order to install a new app you should always use syncdb first, then apply south for migrations if you have any. So, yes, you are not going to have any problem.

migration files and schema

So my Mac went down and now trying to build my project on my new Mac. I need to know when running rake db:migrate does it look at the schema.rb file?
As I am getting
Mysql::Error: Table 'myproject_development.users' doesn't exist: SHOW FIELDS FROM `users`
Even when I run rake db:migrate:up VERSION=001 which has no reference to users I get the same error?
If it matters my migrations start like
001_...
002_...
003_...
20100222171241_...
The migration code generally doesn't look at the schema file. It looks at the names of all the migration files and the database table called schema_migrations, and determines which migrations haven't been run yet. (I believe it does dump the schema at the end of running migrations.) Either your missing a migration, or your schema_migrations table is out of sync with your database.
After making a backup, you can start to troubleshoot. Do you have a migration that creates the users table? If not, where did it go? If you have it, why isn't it running?
FYI The migrations that start with numbers (eg. 001) are older versions of migrations. Sometime around 2.2 or 2.3 the names of the migrations were changed to dates, which is what you are seeing in the later migrations.
There may be a problem with the way those first few migrations are named, and they are not being found (I can't remember the migration story when this naming scheme switched). And maybe it tracked the migrations differently back then so there may be some futzing to make it work with the more modern scheme. You can fairly safely rename them following the new scheme, using the datestamp of the file.
Hope this helps.

How can you get a sql script of changes from Rails migrations (for MySQL)?

I've seen http://github.com/muness/migration_sql_generator, but it doesn't work properly with MySQL for certain important operations. Is there any other way we can capture the sql generated during a rails migration?
The reason I'm asking is that I cannot run the migrations on the production server since it is maintained by technical support (and never touched by developers) at my company. Developers provide a JRuby on Rails war file to tech support and they deploy it via Tomcat. But convincing tech support to install JRuby and Rails just to run the migrations on production is definitely not going to be easy. We want to keep the deployment really simple and with as few dependencies as possible.
We want to simply provide them with a war file and a sql script with db changes.
You may be able to use some kind of tool to generate a diff of two databases. There's a question on that here.
If you diff your test database with your development database just after running the migration that would probably be the easiest way. It might even be worth adding a rake task to do it; get that rake task to depend on the migration and then you could use the new task instead of rake db:migrate to generate a diff each time you migrate.
I actually ended up creating a rake task which monkey-patched Activerecord's sql execution method to also output all sql to a log file (log/database.log). So, the task could be run right before db:migrate like so: rake db:log db:migrate. After that, you can extract the statements which are relevant and put them into a db/sql_migrations/<migration name>.sql file and have your database admins run that when they're ready.
Thankfully I've changed jobs and don't have to deal with that mess of a process anymore though. :)