We have a fairly large MySQL database with more than a million rows of data with every possible data type.
It is a part of a custom MVC application built more than 5 years ago. We have to migrate it now.
There are a large number of queries and insert statements which we want to replace with JSON based web services so that it can be used with every kind of app/device etc.
A large number of PHP functions have been fused with display logic making it tricky. Also there are also a few MySQL functions in the bundle.
Please share tips/suggestions/tools that would be useful for this migration.
There are some tools which I can suggest you to convert your Mysql database to Pgsql one here is some of them:
pgloader
PostgreSQL Data Wizard
pgloader
dataPro
And based on my experience, I suggest you pgloader to load data from MySQL, SQLite, MS SQL Server, dBase files, CSV files and fixed-width data files, and more. Released under The PostgreSQL Licence..
Related
We are finally moving from Excel and .csv files to databases. Currently, most of my Tableau files are connected to large .csv files (.twbx).
Is there any performance differences between PostgreSQL and MySQL in Tableau? Which would you choose if you were starting from scratch?
Right now, I am using pandas to join files together and creating a new .csv file based on the join.(Example, I take a 10mil row file and drop duplicates and create a primary key, then I join it with the same key on a 5mil row file, then I export the new 'Consolidated' file to .csv and connect Tableau to it. Sometimes the joins are complicated involving dates or times and several columns).
I assume I can create a view in a database and then connect to that view rather than creating a separate file, correct? Each of my files could instead be a separate table which should save space and allow me to query dates rather than reading the whole file into memory with pandas.
Some of the people using the RDMS would be completely new to databases in general (dashboards here are just Excel files, no normalization, formulas in the raw data sheet, etc.. it's a mess) so hopefully either choice has some good documentation to lesson the learning curve (inserting new data and selecting data mainly, not the actual database design).
Both will work fine with Tableau. In fact, Tableau's internal data engine is based on Postgres.
Between the two, I think Postgres is more suitable for a central data warehouse. MySQL doesn’t allow certain SQL methods such as Common Table Expressions and Window Functions.
Also, if you’re already using Pandas, Postgres has a built-in Python extension called PL/Python.
However, if you’re looking to store a small amount of data and get to it really fast without using advanced SQL, MySQL would be a fine choice but Postgres will give you a few more options moving forward.
As stated, either database will work and Tableau is basically agnostic to the type of database that you use. Check out https://www.tableau.com/products/techspecs for a full list of all native (inbuilt & optimized) connections that Tableau Server and Desktop offer. But, if your database isn't on that list you can always connect over ODBC.
Personally, I prefer postgres over mysql (I find it really easy to use psycopg2 to write to postgres from python), but mileage will vary.
I have tried some online converters but they are only converting like 50 records from one table. Is there any tool or a proper way to migrate Visual Foxpro Data into mysql data?
There are a couple of utilities on the Leafe.com site that might help.
VFPData2MariaScript Use this script to create a set of INSERT records that detects the structure of your VFP tables. This works well with the utility of STRU2MYSQL_2.PRG which scripts a CREATE TABLE file. This may work for PostgreSQL as well as MariaDB and MySQL. Note that if your table is really large, you may run into execution limits, or file size limits. There is another utility that opens up a connection from VFP and upsizes the data. This utility just creates a script of INSERT statements that you run through phpMyAdmin or the like. Author: Kevin Cully
I'm importing data from SQL Server to mysql.
Currently I need to migrate a single a table with a simple data structure (no constraints, only values), to mysql. The table is 1.6 milion rows
What is a good way to accomplish this?
(I saw other related topics here, but don't see something that I can work with)
The best approach is to write a program using your favorite language to migrate the data. If you are looking for a tool to accomplish this, have a look at MySQL Migration Toolkit. It migrates data from various data sources to MySQL database.
MySQL Migration Toolkit.
You could use SSIS, it can connect to ODBC data destinations and might be quicker to implement than writing an application from scratch.
We need to redo a database in MySQL that has been already done on Informix, is there a way to migrate not only the schema, but the stored procedures as well?
Thanks.
We have a client whom we built a web application that uses an Informix database. Now the client wants to be able to implement the same software but on multiple closed networks (like 20). Doing this using Informix would be very expensive (20 licences X_X).
So the best approach is to redo the database on something like MySQL.
The application was done using Flex, .Net (using ODBC) and Informix.
I have done a similar thing, but I migrated Informix database to PostgreSQL. At first I dbexported the whole database, so the whole data and schema info was in text. Then I wrote some Python programs that translated schema, for example Informix DATETIME YEAR TO SECOND must be converted to timestamp with time zone.
When all CREATE TABLE/INDEX etc worked then I translated .unl files to PostgreSQL COPY commands. You should search how to do bulk load in MySQL, or convert those files to INSERT commands.
After that I started converting stored procedures. While PostgreSQL PL/pgSQL and Informix SPL are very different this part was the hardest and I was able to automatically convert only function "prototypes". Functions body had to be converted manually.
If you completed this you will have to check if your application work well with a new SQL implementation.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I've been banging my head against SQL Server 2005 trying to get a lot of data out. I've been given a database with nearly 300 tables in it and I need to turn this into a MySQL database. My first call was to use bcp but unfortunately it doesn't produce valid CSV - strings aren't encapsulated, so you can't deal with any row that has a string with a comma in it (or whatever you use as a delimiter) and I would still have to hand write all of the create table statements, as obviously CSV doesn't tell you anything about the data types.
What would be better is if there was some tool that could connect to both SQL Server and MySQL, then do a copy. You lose views, stored procedures, trigger, etc, but it isn't hard to copy a table that only uses base types from one DB to another... is it?
Does anybody know of such a tool? I don't mind how many assumptions it makes or what simplifications occur, as long as it supports integer, float, datetime and string. I have to do a lot of pruning, normalising, etc. anyway so I don't care about keeping keys, relationships or anything like that, but I need the initial set of data in fast!
The best way that I have found is the MySQL Migration Toolkit provided by MySQL. I have used it successfully for some large migration projects.
Using MSSQL Management Studio i've transitioned tables with the MySQL OLE DB. Right click on your database and go to "Tasks->Export Data" from there you can specify a MsSQL OLE DB source, the MySQL OLE DB source and create the column mappings between the two data sources.
You'll most likely want to setup the database and tables in advance on the MySQL destination (the export will want to create the tables automatically, but this often results in failure). You can quickly create the tables in MySQL using the "Tasks->Generate Scripts" by right clicking on the database. Once your creation scripts are generated you'll need to step through and search/replace for keywords and types that exist in MSSQL to MYSQL.
Of course you could also backup the database like normal and find a utility which will restore the MSSQL backup on MYSQL. I'm not sure if one exists however.
SQL Server 2005 "Standard", "Developer" and "Enterprise" editions have SSIS, which replaced DTS from SQL server 2000. SSIS has a built-in connection to its own DB, and you can find a connection that someone else has written for MySQL. Here is one example. Once you have your connections, you should be able to create an SSIS package that moves data between the two.
I ddin't have to move data from SQLServer to MySQL, but I imagine that once the MySQL connection is installed, it works the same as moving data between two SQLServer DBs, which is pretty straight forward.
Rolling your own PHP solution will certainly work though I'm not sure if there is a good way to automatically duplicate the schema from one DB to the other (maybe this was your question).
If you are just copying data, and/or you need custom code anyway to convert between modified schemas between the two DB's, I would recommend using PHP 5.2+ and the PDO libraries. You'll be able to connect using PDO ODBC (and use MSSQL drivers). I had a lot of problems getting large text fields and multi-byte characters from MSSQL into PHP using other libraries.
Another tool to try would be the SQLMaestro suite. It is a little tricky nailing down the precise tool, but they have a variety of tools, both free and for purchase that handle a wide variety of tasks for multiple database platforms. I'd suggest trying the Data Wizard tool first for MySQL, since I believe that will have the proper "import" tool you need.