How can I easily convert a Django app from mySQL to PostgreSQL? - mysql

Has anyone done this? Is it an easy process? We're thinking of switching over for transactions and because mysql seems to be "crapping out" lately.

Converting MySQL database to Postgres database with Django
First backup your data of the old Mysql database in json fixtures:
$ python manage.py dumpdata contenttypes --indent=4 --natural-foreign > contenttype.json
$ python manage.py dumpdata --exclude contenttypes --indent=4 --natural-foreign > everything_else.json
Then switch your settings.DATABASES to postgres settings.
Create the tables in Postgresql:
$ python manage.py migrate
Now delete all the content that is automatically made in the migrate (django contenttypes, usergroups etc):
$ python manage.py sqlflush | ./manage.py dbshell
And now you can safely import everything, and keep your pk's the same!
$ python manage.py loaddata contenttype.json
$ python manage.py loaddata everything_else.json
Tested with Django==1.8

I just used this tool to migrate an internal app and it worked wonderfully. https://github.com/maxlapshin/mysql2postgres

You can do that using Django serializers to output the data from MySQL's format into JSON and then back into Postgres. There are some good artices on internet about that:
Migrating Django from MySQL to PostgreSQL the Easy Way
Move a Django site to PostgreSQL: check

I've never done it personally, but it seems like a combination of the dumpdata and loaddata options of manage.py would be able to solve your problem quite easily. That is, unless you have a lot of database-specific things living outside the ORM such as stored procedures.

I've not done it either.
I'd first follow this migration guide, there is a MySql section which should take care of all your data. Then django just switch the mysql to postgre in the settings. I think that should be ok.
I found another question on stackoverflow which should help with the converting mysql to postgre here.

python manage.py dump.data >> data.json
Create database and user in postrgesql
Set your just created database in postrgesql as default database in django settings or use param --database=your_postrgesql_database next steps
Run syncdb for create tables.
python syncdb [--database=your_postrgesql_database] --noinput
Create dump without data, drop all tables and load dump. Or truncate all tables (table django_content_type whith data which can be not equals your old data - it is way to many errors). At this step we need empty tables in postgresql-db.
When you have empty tables in postgresql-db just load your data:
python manage.py loaddata data.json
And be fun!

I wrote a Django management command that copies one database to another:
https://gist.github.com/mturilin/1ed9763ab4aa98516a7d
You need to add both database in the settings and use this command:
./manage.py copy_db from_database to_database app1 app2 app3 --delete --ignore-errors
What cool about this command is that it recursively copy dependent objects. For example, if the model have 2 foreign keys and two Many-to-Many relationships, it will copy the other objects first to ensure you won't get foreign key violation error.

Related

How to Load a .dump File Format Using manage.py loaddata in Django?

I have a data.dump file for mysql and am trying to use manage.py loaddatato fill my db with this data file, but getting this error:
CommandError: Problem installing fixture '˜/Downloads/data': dump is not a known serialization format.
Apparently this is not a known format for Django loaddata. The question is how I can convert a .dump to .json, so I can use it with loaddata command?
If you crated your data.dump via MySQL there is, AFAIK, no way to load it via manage.py loaddata.
You can use this command to import so-called "fixtures" created with manage.py dumpdata
You will have to import your MySQL data dump directly into your new MySQL database.

How to fix 'Django: table doesn't exist' when migrating from someone else's project?

I get the following error while trying to run the command python manage.py makemigrations invigilators:
django.db.utils.ProgrammingError: (1146, "Table 'mydatabase.invigilators_shift' doesn't exist")
class Shift(models.Model):
shiftName = models.CharField(max_length=255,blank=False,unique=True)
exam = models.ForeignKey(Exam,on_delete=models.CASCADE,related_name="shifts")
startTime = models.TimeField()
endTime = models.TimeField()
def __str__(self):
return self.shiftName
I have already cleared all previous migrations and an empty database is already created.
Try running these commands
python manage.py migrate <app_name> zero
delete all initial.py file under app migrations
python manage.py makemigrations
python manage.py migrate
In my case I get the table doesn't exist errors from forms.py. I then comment out the entries in forms.py that refer to a database table. Then I can run makemigrations and migrate. Then uncomment the lines and everything works. This has plagued me for multiple Django releases.
Below steps solved the problem for me
Delete all migrations files
python manage.py makemigrations
(Create your initial migration file 0001_inital.py)
python manage.py migrate --fake <app_name> zero
( Tell Django to mark the migrations as having been applied or unapplied, but without actually running the SQL to change your database schema. This brings the migrations to zeroth state. Otherwise Django will think there's no change to apply )
python manage.py migrate
First Step:
Just "Cut" The all models from Models.py & paste that models to the any other text file or notepad.
Second Step:
Just "Cut" the all forms from forms.py & paste at the the same text file at you pasted the Models.
Third Step:
Make a Comment of all imported models & forms in views.py Because we don't want errors.
Fourth Step:
After these three steps you have to just run the command "python manage.py migrate".
Fifth Step:
After that command You have to just bring back all the models in models.py and all the forms in forms.py & uncomment all the imported forms and models of views.py.
Last Step:
After these all steps you have to just run another two commands
1."python manage.py makemigrations"
2."python manage.py migrate"
& Done,It works.

Yii2 run only specific migration

In Yii2, when I use the
yii migrate
command, I get a long list of remaining migrations. How can I only run one specific migration in the list without performing the others?
Run migrate/history to list migrations have been applied:
./yii migrate/history
Copy the name of migration you want to return to later (lets say it is 'm160101_185401_initial_migration'). Save it somewhere because you're going to need this later.
Mark migration history at the one just before the one you need to run:
./yii migrate/mark m170101_185401_create_news_table
Run one migration:
./yii migrate 1
Reset migration history:
./yii migrate/mark m160101_185401_initial_migration
yii migrate --migrationPath=#app/modules/forum/
If you want to skip some migrations that have been implemented in your database without running the migrations, you set your migrations state without running them.
By "marking" the migrations you also assure they will no longer be re-prompted and will be regarded as "done".
you can read about marking in the Yii docs here
To run specific migration, you can mark(skip) migrations upto just before one you want run.
You can mark migration by using one of following command:
Using timestamp to specify the migration yii migrate/mark 150101_185401
Using a string that can be parsed by strtotime() yii migrate/mark "2015-01-01 18:54:01"
Using full nameyii migrate/mark m150101_185401_create_news_table
Using UNIX timestamp yii migrate/mark 1392853618

How to export subset of data from mongodb

I currently have a large database, and I need a means of backing up subsets of the data that can then be imported on another mongodb instance.
For example, I would need to find all documents that contain a key, so essentially: find({key: 'somekey'}), and then export that data set. I thought to simply run the query in NodeJS, and save the data in JSON format. I don't think this is optimal as through my understanding simply importing the JSON data again (if needed in the future) won't be a straightforward task as the data-types will be lost.
So my question is, how would I go about exporting a subset of the dataset so that it may be possibly re-imported into another mongodb instance on another server.
Thanks to #Veeram's comment, the way to do this is as BSON so that it retains all the data structure:
sudo mongodump -d DB_Name -c Collection -q '{"key_name": "value"}' --out /home/collection
Then to import it back:
sudo mongorestore -d DB_Name -c Collection /home/collection/DB_Name/Collection.bson

Problems with contenttypes when loading a fixture in Django

I am having trouble loading Django fixtures into my MySQL database because of contenttypes conflicts. First I tried dumping the data from only my app like this:
./manage.py dumpdata escola > fixture.json
but I kept getting missing foreign key problems, because my app "escola" uses tables from other applications. I kept adding additional apps until I got to this:
./manage.py dumpdata contenttypes auth escola > fixture.json
Now the problem is the following constraint violation when I try to load the data as a test fixture:
IntegrityError: (1062, "Duplicate entry 'escola-t23aluno' for key 2")
It seems the problem is that Django is trying to dynamically recreate contenttypes with different primary key values that conflict with the primary key values from the fixture. This appears to be the same as bug documented here: http://code.djangoproject.com/ticket/7052
The problem is that the recommended workaround is to dump the contenttypes app which I'm already doing!? What gives? If it makes any difference I do have some custom model permissions as documented here: http://docs.djangoproject.com/en/dev/ref/models/options/#permissions
manage.py dumpdata --natural will use a more durable representation of foreign keys. In django they are called "natural keys". For example:
Permission.codename is used in favour of Permission.id
User.username is used in favour of User.id
Read more: natural keys section in "serializing django objects"
Some other useful arguments for dumpdata:
--indent=4 make it human readable.
-e sessions exclude session data
-e admin exclude history of admin actions on admin site
-e contenttypes -e auth.Permission exclude objects which are recreated automatically from schema every time during syncdb. Only use it together with --natural or else you might end up with badly aligned id numbers.
The answers here all old... As of 2017, the best answer is:
manage.py dumpdata --natural-foreign --natural-primary -e contenttypes -e auth.Permission --indent 4
Yes, this is really irritating. For a while I worked around it by doing a "manage.py reset" on the contenttypes app prior to loading the fixture (to get rid of the automatically-generated contenttypes data that differed from the dumped version). That worked, but eventually I got sick of the hassles and abandoned fixtures entirely in favor of straight SQL dumps (of course, then you lose DB portability).
update - the best answer is to use the --natural flag to dumpdata, as noted in an answer below. That flag did not exist yet when I wrote this answer.
Try skipping contenttypes when creating fixture:
./manage.py dumpdata --exclude contenttypes > fixture.json
It worked for me in a similar situation for unit tests, your insight regarding the contenttypes really helped!
I was not using MySQL but instead importing some data from a live server into sqlite. Clearing the contenttypes app data before performing loaddata did the trick:
from django.contrib.contenttypes.models import ContentType
ContentType.objects.all().delete()
quit()
And then
python manage.py loaddata data.json
I have resolved this issue in my test cases by resetting the contenttypes app from the unit test prior to loading my dump file. Carl suggested this already using the manage.py command and I do the same thing only using the call_command method:
>>> from django.core import management
>>> management.call_command("flush", verbosity=0, interactive=False)
>>> management.call_command("reset", "contenttypes", verbosity=0, interactive=False)
>>> management.call_command("loaddata", "full_test_data.json", verbosity=0)
My full_test_data.json fixture contains the contenttypes app dump that corresponds to the rest of the test data. By resetting the app before loading, it prevents the duplicate key IntegrityError.
You need to use natural keys to represent any foreign key and many-to-many relationships. Moreover, it might be a good idea to exclude the session table in the sessions app, and the logentry table in the admin app.
Django 1.7+
python manage.py dumpdata --natural-foreign --exclude contenttypes --exclude auth.permission --exclude admin.logentry --exclude sessions.session --indent 4 > fixture.json
Django <1.7
python manage.py dumpdata --natural --exclude contenttypes --exclude auth.permission --exclude admin.logentry --exclude sessions.session --indent 4 > fixture.json
According to the Django documentation, --natural has been deprecated in version 1.7, so the option --natural-foreign should be used instead.
You can also omit the primary key in the serialized data of this object since it can be calculated during deserialization by passing the --natural-primary flag.
python manage.py dumpdata --natural-foreign --natural-primary --exclude contenttypes --exclude auth.permission --exclude admin.logentry --exclude sessions.session --indent 4 > fixture.json
python manage.py dumpdata --natural-primary --exclude=contenttypes --exclude=auth.Permission --exclude=admin.logentry --exclude=sessions.session --indent 4 > initial_data.json
This works for me. Here I am excluding everything bubt the actual models.
If you see any other model other than the models that you created you can safely exclude those. One drawback of this approach is you loose on log data as well as auth data.
./manage.py dumpdata app.Model --natural-foreign
will change
"content_type": 123
to
"content_type": [
"app_label",
"model"
],
And fixture works for TestCase now
Django 2.2.5
python manage.py dumpdata --exclude=contenttypes > datadump.json
it helped me
It's really, really annoying .. I get bitten by this every single time.
I tried to dumpdata with --exclude contenttypes and --natural, I always get problems..
What works best for me is simply doing a truncate table django_content_type; after the syncdb and THEN load the data.
Of course for initial_data.json autoloading you're fallball.
I'm going to give another possible answer that I just figured out. Maybe it'll help the OP, maybe it'll help somebody else.
I've got a many-to-many relationship table. It has a primary key and the two foreign keys to the other tables. I found that if I have an entry in the fixture whose two foreign keys are the same as another entry already in the table with a different pk, it will fail. M2M relationship tables have a "unique together" for the two foreign keys.
So, if it's a M2M relationship that is breaking, look at the foreign keys it's adding, look at your database to see if that pair of FKs are already listed under a different PK.
I had encountered similar error sometimes ago. It turned out that I was trying to load the fixtures before creating the necessary tables. So I did:
$ python manage.py makemigrations
$ python manage.py migrate
$ python manage.py loaddata fixtures/initial_data.json
And it worked like a charm
I tried every method from above, Nothing worked for me. I have to exclude the complete auth model and works fine.
python manage.py dumpdata --natural-primary --exclude=contenttypes --exclude=auth --exclude=admin.logentry --exclude=sessions.session --indent 4 > live.json
In my case I had dumped the data from auth (./manage.py dumpddata auth > fixtures/auth.json) to use the fixture for testing purposes.
The development continued and I removed most of the models I had defined in models.py and this is when I started to see this annoying problem.
My solution was regenerating the auth.json fixture again. This one had removed lots of entries in auth.permission related to the old models I had.
I've fixed this by adding in my tests setUp and tearDown
from django.core import management
=====
def setUp(self):
management.call_command("loaddata", "all-data.yaml", verbosity=0)
super(login_page_test, self).setUp()
def tearDown(self):
management.call_command("flush", verbosity=0, interactive=False)
super(login_page_test, self).setUp()
I used pgloader, just take a few seconds to migrate successfully:
$ pgloader project.load
project.load file with:
load database
from sqlite:////path/to/dev.db
into postgresql://user:pwd#localhost/db_name
with include drop, create tables, create indexes, reset sequences
set work_mem to '16MB', maintenance_work_mem to '512 MB';