converting a project and looking for a way to connect to DB schema, read it and generate knexJS migration files to integrate in new project.
Is there a way to do this? and how?
No, there is no way to extract migration file for current schema of some database. If you really want to create one, it must be written manually.
But you don't even need one. You can just start using knex, with your old database dump and in the future when there should be changes done to the schema you can write new migrations for those specific incremental changes to the schema.
For running migrations you should use knex's migration functionality.
Related
We are migrating our existing code base to a new robust code base for better performance and reliability. We have a MySQL database with current data. Now we have modified our entities in our spring boot application which will change our schema for the new database structure. I am in search of a tool which will help me migrate all the data from the old MySQL database to a newly created MySQL database with changes according to the latest schema design. I think I will have to write some code to match the new database architecture as no tool will do that refactoring according to my requirement. What tool should be helpful to achieve this?
Footnotes:
I am working in a microservice architecture.
I have integrated liquibase with maven plugin support.
I have seen Apache Spark and ETL, but they need
Provide your feedback if you have any relative experience.
We have done the migration of near about 3000 users data from 160(SQL) to near about 75 tables (MySQL) as per new schema.
As per our experience, I can suggest following things -
1. Prepare mapping sheet of the column in the table to table manner.
2. Migrate them into temporary tables.
3. Test the temporary table's data with the old one. Compare data of each table column by column. You can use ETL tool or excel for comparison.
4. Then write down sp or script for actual migration if no bugs found.
I created some tables using rails. Now I want to modify the structure of a few of them. I know it can be done using rails migration. But i was wondering if it would cause any anomaly in the rails app if I modify the schemas using mysql rdbms?
Doing such changes through a migration has the advantage of not losing the changes if you decide to recreate/remigrate the database.
Also it serves as documentation. Imagine if your coworker altered some tables sneakily (and then you both forgot about it).
Technically, updating schemas directly in the database should work, but don't do it.
To add to Sergio's point, you're missing a simple fact - Rails' migrations create the famous db/schema.rb file - from which your migrations pull all their data.
The importance of schema.rb is overlooked - it is one of the most crucial aspects of your application.
db/schema.rb
The schema gives all your migrations a version of your DB to change / add to. Each time you perform a migration, Rails changes the schema file to ensure it has a "blueprint" of your db stored on file.
The schema is then able to rebuild the database using such methods as rake db:schema:load (ONLY RECOMMENDED FOR NEW INSTALLS -- DELETES PREVIOUS DATA)
So whilst there's no problem setting up the db using the db's own native tools, I recommend against it. You need to keep your migrations up to speed so that Rails can build the appropriate tables from its schema.
Is it possible to build a seed data file from an existing mysql database?
I know how to populate a new database with all the tables from an existing database. I'm curious if its possible to build a seed file (seed.rb) from a db.
Try this https://github.com/ludicast/yaml_db
Should provide all the abilities you're looking for.
There are a few gems for the purpose of extracting seeds from an existing database. You could try seed_dump, for instance.
I'm considering a MySQL to Postgresql migration for my web application, but I'm having a really hard time converting my existing MySQL database to Postgresql.
I tried :
mysldump with --compatible=postgresql
migration wizard from EnterpriseDB
Postgresql Data Wizard from EMS
DBConvert from DMSoft
and NONE of the above programs do a good job converting my database!
I saw some Perl and Python scripts for converting mysql to postgresql, but I can't figure out how to use them....(I installed ActivePerl and don't understand what I'm supposed to do next to run that script!)
I use Auto Increment fields (as a primary key) all the time, and these are just ignored... I understand that Postgresql does auto-increments in another way (with sequences), but it can't be THAT hard for MIGRATION software to implement that, or is it?
Did anybody have better luck converting a MySQL database with auto-increments as primary keys?
I know this is probably not the answer you are looking for, but: I don't believe in "automated" migration tools.
Take your existing SQL Scripts that create your database schema, do a search and replace for the necessary data types (autonumber maps to serial which does all the sequence handling automagically for you), remove all the "engine=" stuff and then run the new script against Postgres.
Dump the old database into flat files and import them into the target.
I have done this several times with sample databases that were intended for MySQL and it really doesn't take that long.
Probably just as long as trying all the different "automated" tools.
Why not use an ETL Tool? you dont have to worry about dumps or stuff like that.
I have migrated to PostgresSQL and MySQL and have had no problems with the auto increment fields.
You just need to know the connection credentials and thats it. I personally use Pentaho ( it's open source ).
Download Pentaho ETL from http://kettle.pentaho.org/
Unzip and run Pentaho (using .bat file spoon.bat)
Create a new Job:
Create DB connection for source data base (PostgreSQL) - using menu: Tools→Wizard→Create DataBase Connection (F3) Create DB connection for destination data base (Mysql) - using technique described above.
Run the Wizard: Tools → Wizard → Copy Tables (Ctrl-F10).
Select source (left dialog panel), and destination (left dialog panel). Click Finish.
The Job will be generated - Run the job.
If you need any help let me know.
Even when you familiar with all "PostgreSQL gotchas", doing every step by hand may take a lot of time, especially when your db is "big".
Try some other scripts/tools.
I know this is an old question but I just ran into the same problem migrating from MySQL to Postgres. After trying several migration tools out the very best one I could find, which will migrate your database structure as cleanly as possible, was Pgloader https://github.com/dimitri/pgloader/ it will take care of changing the Auto Increment to Postgres sequences no problem and it's super fast.
how to migrate Grail's HSQLDB embedded database(That contains my App's Data that I don't want to lose) into external one, such as MySQL or ApacheDerby?
If your data isn't important just let hibernate regenerate your schema, else try this: http://www.grails.org/plugin/liquibase
The MySQL Migration Toolkit may be exactly what you need.
A little searching turned up this article that shows examples of what the GUI Tool looks like.
I haven't had to migrate data from a HSQLDB to any other DB, but if I had data that I didn't want to use in a HSQLDB then I'd definitely try this method.
You are going to want to backup the
HSQLDB database that you want to save
(I'm assuming you used a file DB
rather than an in-memory one right?)
Change your DataSource to a MySQL
datasource with the dbCreate set to
update (or something non-destructive)
Run the Migration Toolkit and migrate
your data
Otherwise you ought to be able to view the data in your DB by using another tool (DBVisualizer, RazorDB, or others) and they might be able to help you export the data.
Because grails uses hibernate underneath, no migration is necessary. All you need to do is repoint your conf\DataSources.groovy to the new database, and next time you startup, it will create tables in the new DB.
See section 3.3 in this doc for more information on MySQL config.