I have this requirement to migrate a live SQL server 2008 to MySQL server 5, I have tried the MySQL migration toolkit but its throwing lots of migration errors. Someone here in Stackoverflow suggested that I use Talend. I installed Talend Open Studio for Data Integration but its quite a general purpose tool that there's no quickstart to do database migration like what I really needed.
Where can I find a quickstart guide to use Talend for such job?
Talend is quite generic data integration tool, but you should get the job done by following these steps (which actually apply to any database-types: PostgreSQL, Oracle, etc.):
Create a new database connection in Repository-panel, to the Metadata-"Db Connections"-node. Name it and select "DB Type" as "Microsoft SQL Server" (or whatever is your source database) and give the connection details.
Create another new database connection, but this time as "MySQL" (or whatever is your target database).
Create a new job to "Job designs" (in Repository).
Expand the "SQL Server"-DB-connection from the Metadata, and drag the desired table (under "Table schemas") to the work-area of the job AND select tMSSqlInput.
Drag the MySQL-DB-connection from the Metadata to the work-area AND select tMysqlOutput. Define the target table.
Click Palette and drag tMap component to the work-area between the two previous components.
Connect tMSSqlInput to tMap and tMap to tMysqlOutput with arrows. You do that by right-clicking the tMSSqlInput, selecting Row-Main and dragging the arrow to the tMap. Same from tMap to tMysqlOutput.
Double-click the tMap, select all columns from the left and drag them to the right. The tMap component allows you to make data transformations. It might be that you wouldn't even need to use the tMap between the database-components. But it is good to have just in case.
Run the job, and troubleshoot as necessary. If something goes wonky, restart Talend Studio or just close and open the job, in many occasions it helps.
The MySQL Migration Toolkit is deprecated long time ago. You should try the new Migration Wizard that comes with MySQl Workbench 5.2.41. It will help you to migrate both the tables and their data. You can have an overview of its capabilities here:
How-To: Guide to Database Migration from Microsoft SQL Server using MySQL Workbench
If you need to get started with Talend, I can suggest you check our documentation page http://www.talend.com/resources/documentation.php , where you will be able to find installation guides and user manuals.
Also, there's an interesting section on our website where you will be able to check out our tutorials, that should help out: http://www.talendforge.org/tutorials/menu.php
xybrek, unfortunately there's no tutorial that covers your specific case exactly.
Download the manuals that Bart linked to and start trying things out in Talend.
The components you'll be looking at are:
tMssqlInput - reads a table in a MsSQL db
tMySQLOutput - directs a flow of data, say the output of the mssqlinput component, into a mysql table
tMySQLRow - lets you execute a query on a mysql table/database
And possibly others. These are just some pointers to the more obvious components.
You can read more about those and see examples in the Reference Guide manual which covers all components in Talend.
Good luck!
Related
I have one MS SQL DB (very big) and MySQL. I must move data from MS SQL DB to MySQL by doing SQL. It mean that source data & destination data arenot same structure. When source data is updated, the updating is reflected into destionation. Please give me an advice which is the suitable way?
Using SSIS
Using store procedure & write my own script
Other way.
Any advice is highly appreciated.
Cleanest way will be using stored procedures since it is not a complete immigration. By doing this you will have everything under control. MSSQL has a nice feature linked servers, a bit slow but worths it in most of cases. You can find instructions here: http://www.sqlservercentral.com/Forums/Topic340912-146-1.aspx
An example:
UPDATE your_mysql_database.dbo.your_mysql_table SET col1=...
To migrate from MSSQL to MYSQL, Please use MYSQL Migration Toolkit
To download and use the MYSQL Migration Toolkit, Please use the following link http://dev.mysql.com/downloads/gui-tools/5.0.html
The MySQL Workbench Migration Wizard is designed to save DBA and developer time by providing visual, point and click ease of use around all phases of configuring and managing a complex migration process:
Database migrations - enables migrations from Microsoft SQL Server, PostgreSQL, Sybase ASE, Sybase SQL Anywhere, SQLite, and more.
Migration project management - allows migrations to be configured, copied, edited, executed and scheduled.
Source and Target selection - allows users to define specific data sources and to analyze source data in advance of the migration.
Object migration - allows users to select objects to migrate, assign source to target mappings where needed, edit migration scripts and create the target schema.
Data migration - allows users to map source and target data and data types, set up data transfer and assign post data transfer events where needed.
Version Upgrades - using migration users can easily move databases off older MySQL versions to the latest.
I have an .sql script that contains inserts and creates tables. I used the "Create EER Model From Script"
It created the tables but I can't see the data inside these tables.
I went to the query menu and tried to make a query but it gives me an error about not being able to connect to localhost.
Am I doing it right?
As documented under Create EER Model from SQL Script:
Clicking this action item launches the Reverse Engineer SQL Script wizard. This is a multi-stage wizard that enables you to select the script you want to create your model from.
For further information, see Section 7.7.9.1, “Reverse Engineering Using a Create Script”.
Following that link:
However, if you are working with a script that also contains DML statements you need not remove them; they will be ignored.
Instead, you want the Manage Data Import/Export option under Server Administration (within the Workspace section of the Home window).
You are confusing things here. Creating a model from a script is a process where meta data is examined and a model is created that you can then use to modify your schema structure, further design your db objects and all that. Modeling is a design process for the structure of your schema/db so it only deals with meta data. It's also used for documentation (e.g. in teams).
On the other hand there's normal sql work with existing db objects and/or actually creating/deleting/modifying db objects. In order to do the latter you must have an understanding of the design of the schema (which you could get by using the modeling part of MySQL Workbench, but not only by that). This is also the place to load a script, run it to insert data and such.
The error you mentioned regarding the connection is yet another problem and you need to solve this first to be able to even access your server. And yes, you have to install a server first somewhere. MySQL Workbench is a tool to visually work with your server(s) in opposition to the MySQL command line client which is a pure text interface (but still also a client application for your MySQL servers).
If you are on Windows and want a MySQL server installed locally (e.g. for testing) your best option is to download the MySQL Installer which greatly simplifies installing any of the tools from the MySQL family (server, client tools, connectors, documentation and more).
I want to take the whole database. Where do I find the database file?
And is there a way to write the whole database with all data to a text file (like the one in SQL Server)?
How to generate SQL scripts for your database in Workbench
In Workbench Central (the default "Home" tab) connect to your MySQL instance, opening a SQL Editor tab.
Click on the SQL Editor tab and select your database from the SCHEMAS list in the Object Browser on the left.
From the menu select Database > Reverse Engineer and follow the prompts. The wizard will lead you through connecting to your instance, selecting your database, and choosing the types of objects you want to reverse engineer.
When you're all done, you will have at least one new tab called MySQL Model. You may also have a tab called EER Diagram which is cool but not relevant here.
Click in the MySQL Model tab
Select Database > Forward Engineer
Follow the prompts. Many options present themselves, including Generate INSERT Scripts for Tables which allows you to script out the data contained within your tables (perfect for lookup tables).
Soon you will see the generated script in front of you. At this point you can Copy to Clipboard or Save to Text File.
The wizard will take you further, but if you just want the script you can stop here.
A word of caution: the scripts are generated with CREATE commands. If you want ALTER you'll have to (as far as I can tell) manually change the CREATEs to ALTERs.
This is guaranteed to work, I just did it tonight.
Q#1: I would guess that it's somewhere on your MySQL server?
Q#2: Yes, this is possible. You have to establish a connection via Server Administration. There you can clone any table or the entire database.
This tutorial might be useful.
EDIT
Since the provided link is no longer active, here's a SO answer outlining the process of creating a DB backup in Workbench.
In MySQL Workbench 6, commands have been repositioned as the "Server Administration" tab is gone.
You now find the option "Data Export" under the "Management" section when you open a standard server connection.
there is data export option in MySQL workbech
I found this question by searching Google for "mysql workbench export database sql file". The answers here did not help me, but I eventually did find the answer, so I am posting it here for future generations to find:
Answer
In MySQLWorkbench 6.0, do the following:
Select the appropriate database under MySQL Connections
On the top-left hand side of screen, under the MANAGEMENT heading, select "Data Export".
Here is a screenshot for reference:
None of these worked for me. I'm using Mac OS 10.10.5 and Workbench 6.3. What worked for me is Database->Migration Wizard... Flow the steps very carefully
In the top menu of MySQL Workbench click on database and then on forward engineer. In the options menu with which you will be presented, make sure to have "generate insert statements for tables" set.
Try the export function of phpMyAdmin.
I think there is also a possibility to copy the database files from one server to another, but I do not have a server available at the moment so I can't test it.
Using Windows 10 and MySql Workbench 8.0
Go to Server tab
Go to Database Export
This opens up something like this
Select the schema to export in the Tables to export
Click the button Start Export
Surprisingly the Data Export in the MySql Workbench is not just for data, in fact it is ideal for generating SQL scripts for the whole database (including views, stored procedures and functions) with just a few clicks. If you want just the scripts and no data simply select the "Skip table data" option. It can generate separate files or a self contained file. Here are more details about the feature: http://dev.mysql.com/doc/workbench/en/wb-mysql-connections-navigator-management-data-export.html
in mysql workbench server>>>>>>export Data
then follow instructions it will generate insert statements for all tables data each table will has .sql file for all its contained data
I'm considering a MySQL to Postgresql migration for my web application, but I'm having a really hard time converting my existing MySQL database to Postgresql.
I tried :
mysldump with --compatible=postgresql
migration wizard from EnterpriseDB
Postgresql Data Wizard from EMS
DBConvert from DMSoft
and NONE of the above programs do a good job converting my database!
I saw some Perl and Python scripts for converting mysql to postgresql, but I can't figure out how to use them....(I installed ActivePerl and don't understand what I'm supposed to do next to run that script!)
I use Auto Increment fields (as a primary key) all the time, and these are just ignored... I understand that Postgresql does auto-increments in another way (with sequences), but it can't be THAT hard for MIGRATION software to implement that, or is it?
Did anybody have better luck converting a MySQL database with auto-increments as primary keys?
I know this is probably not the answer you are looking for, but: I don't believe in "automated" migration tools.
Take your existing SQL Scripts that create your database schema, do a search and replace for the necessary data types (autonumber maps to serial which does all the sequence handling automagically for you), remove all the "engine=" stuff and then run the new script against Postgres.
Dump the old database into flat files and import them into the target.
I have done this several times with sample databases that were intended for MySQL and it really doesn't take that long.
Probably just as long as trying all the different "automated" tools.
Why not use an ETL Tool? you dont have to worry about dumps or stuff like that.
I have migrated to PostgresSQL and MySQL and have had no problems with the auto increment fields.
You just need to know the connection credentials and thats it. I personally use Pentaho ( it's open source ).
Download Pentaho ETL from http://kettle.pentaho.org/
Unzip and run Pentaho (using .bat file spoon.bat)
Create a new Job:
Create DB connection for source data base (PostgreSQL) - using menu: Tools→Wizard→Create DataBase Connection (F3) Create DB connection for destination data base (Mysql) - using technique described above.
Run the Wizard: Tools → Wizard → Copy Tables (Ctrl-F10).
Select source (left dialog panel), and destination (left dialog panel). Click Finish.
The Job will be generated - Run the job.
If you need any help let me know.
Even when you familiar with all "PostgreSQL gotchas", doing every step by hand may take a lot of time, especially when your db is "big".
Try some other scripts/tools.
I know this is an old question but I just ran into the same problem migrating from MySQL to Postgres. After trying several migration tools out the very best one I could find, which will migrate your database structure as cleanly as possible, was Pgloader https://github.com/dimitri/pgloader/ it will take care of changing the Auto Increment to Postgres sequences no problem and it's super fast.
I'm creating a new Asp.Net MVC 3 application. Visual Studio does a lot of the job of create the database and initial layout. Very nice! I will upload that initial files to my server, but I want that it runs using the MySql database on the server.
There's some quick/easy way to do it? I'm not worried about the data, just the structure of the tables, and the connection/configuration changes.
Thank you very much!
You can export any MS-SQL database as a Script (Sql Server manager).
Fix it up to make it compatible.
But you will also need a Membership provider, look around if there exist any for MySql, otherwise you'll have to create one (movie).
There are a number of tools listed in "Migrating from Microsoft SQL Server and Access to MySQL".
Or (assuming that you're using column types that exist on both platforms) you can write a script to convert a schema dump from SqlServer into MySQL (or do the conversion by hand in a text editor). Even better yet, you can write a program program to read the INFORMATION_SCHEMA table from SqlServer and produce the necessary CREATE TABLE... statements in mysql. Lots of options.