We are in the process of moving our backend from ms sql server to mysql. Actually we currently use a couple mysql servers, but mostly ms sql server. I mention this because we are not totally new to mysql. Each day we do a lot of ETL to keep our backend in sync with a legacy system. We move a lot of data and working with sql server has been so much easier than working with mysql for ETL. I know SSIS is MS, but still it has been a headache.
We are using sql server 2012 and BIDS 2010. It has been a struggle to move mysql data at the same rate as ms sql data. We are mainly dealing with innodb tables in mysql. To summarize I have been using the mysql ODBC connector and the ODBC destination in SSIS. The first step is to turn autocommit off on imports. Even with that setting off I can see in package execution that the data source ends up waiting on the destination. It gets about 40,000 rows ahead and waits.
Next I export the data to a text file and then import using a sql task and the INFILE command. This gives pretty good performance, but at the expense of more moving parts. I've had a couple issues with this approach, but it does work and perform well.
Lastly I tried a 3rd party SSIS component from Devart. It creates custom mysql source and destination components. The performance isn't as good as INFILE, but it's not bad and it makes the package simple like when dealing with sql server... a data source and a data destination. No messing with auto commits, exports, INFILE, etc. However I can't use the connections to do other tasks like truncate tables and stuff. So I still have my ODBC connection to do those tasks. I'm going to ask Devart about this.
Right now it looks like Devart is going to be a nice balance. If I absolutely need the performance I have the INFILE method.
I also tried the mysql net connector and could not get that to work at all. I'm running on Windows 7 64bit with Sql Server 2012 64bit. Basically everything I need in BIDS runs in 32bit so I'm guessing this part of issues.
My question is what are others doing when it comes to moving mysql data with SSIS? It has been such a hassle. It would be nice to get some input on what others are doing. What methods are you using? Are you using 3rd party components? Is there a better/dedicated place to discuss SSIS and mysql?
Related
I am using a MS Access 2013 light application that was developed by a third party. I did not do the coding/design/management of the project, but I am responsible for implementation for my team. I also do not have the option of switching to another solution, but I do have access to the vba code so I can make tweaks to clean up their mess.
My problem is this:
Set up application with my data (a-ok).
Run the built in, fairly complex third party macro.
For most cases, things are just fine... but when running it on a
larger dataset the filesize of the Access file exceeds 2GB and the
entire operation fails.
On fail, the process has to be restarted. For the same data set,
it fails each and every time it reaches approximately 55% complete.
I am unable to complete my work because of this. :|
Solutions tried:
Compact and repair - Fine when it fully executes, but the issue is that it reaches 2GB while the macro is running and cannot be interrupted.
Splitting the database - Splits OK, but doesn't fix issue.
Attempting to trigger a compact and repair inside the macro during the loop - Fails because Access cannot lock the database.
Desired solution:
A way to prevent the file growth/bloat while the macro is running. Either through a compartmentalization of the process or through some other wizardry I am unaware of at this time.
A solution that does not require extensive reconfiguration of the underlying code. I can deal with inefficient - so long as I can fix this issue for this one instance (1 critical error in 44 runs of different data in the database.
Any help?
I would recommend compact on close for easy dirty solution
On the File tab, click Options.
In the Access Options dialog box, click Current Database.
Under Application Options, select the Compact on Close check box.
ADVANCED SOLUTION
The other solution requires splitting the database.
After splitting you have another option.
Use the front with a sql server (check which version is suitable for you, I think the lite (free) version is enough to start with if you don't expect a hige amount of data)
Split the database
Install sql server (mysql or sql server express edition)
Create all tables in the sql server
Link the front to the sql server
I think davejal pretty much nailed this one.
If you have a handful of really large tables, you can put those into another Access DB, and make a link to those.
https://support.office.com/en-us/article/Import-or-link-to-data-in-another-Access-database-095ab408-89c7-45b3-aac2-58036e45fcf6
The 2GB limit is per DB.
Or, upgrade to SQL Server Express for free, and use Access as a front end to that SQL Server backend.
SQL-Server Backend, MS Access Frontend: Connection
Here's a link to get SQL Server Express.
https://www.microsoft.com/en-us/download/details.aspx?id=42299
We have a database in production running on SQL Server 2012, the database was copied and then changes were made to the schema, structure, relationships during further development of our application that uses the database.
We are now ready to migrate the data from the old production database to our newly developed database. We spent a late night last night trying to figure out how to get that done using Red Gate's SQL Schema Comare, and SQL Data Compare. We were not very successful as kept getting errors during the SQL Schema Compare Deployment.
The options and features of Red Gate's tools are very extensive and I don't have much experience using them. I tried creating migration scripts within SQL Compare, but as the databases weren't linked up to source control prior, I don't see where or how to write these scripts.
The question I have is what is the proper sequence to migrate data using Red Gate's tools, or would it just be easier to create SSIS packages to handle it all. The amount of records we are migrating is relatively low less than 10,000 records. We would like to preserve the identity values as we have some scripts that may depend on the specific identity values.
Any tips, suggestions, and comments are much appreciated.
Thanks
We are planning to migrate an Oracle 10g database to SQL Server 2008 R2. At the moment nothing is implemented in the target database and this will give us the opportunity to change and improve the existing schema during the migration.
Not only the data, but also stored procedures and views have to be imported.
I already worked with SSIS and I found an excellent product for data manipulation. A colleague mentioned SSMA for the migration. However after some research on the net it seems that it would be suitable mainly for data migration and conversion, while SSIS seems to provide a wider set of functinalities (Tasks, custom scripts, etc).
Which are the pro/contra of the two products and which one would best fit for the task?
I would recommend a hybrid approach. Use SSMA to convert the schema and objects from Oracle to SQL Server. Then improve and or change the objects as you see fit on the SQL end. Once your satisfied with your new schema. Use SSIS to move the data still waiting on the Oracle side into the new schema waiting for it on SQL.
As for a quick comparison of SSMA and SSIS... SSIS is by far superior for the ETL aspects of moving data from one place to another; but I wouldn't necessarily recommend it for the creation/modification phase of what you describe above. I think you'll find that process much easier with SSMA. On the flip side SSMA doesn't offer much in the way of transformation during the copy process.
I would go for an hybrid of the two.
Do you know you can trig SSMA from command line? This way you can execute the SSMA migration as a part of the SSIS solution.
You can also save your SSMA project as an SSIS package:
Once migrated keep doing the extra work with SSIS.
I am trying to transfer 67,714,854 rows from MySQL to SQL Server using SSIS. The package times out after transferring 14,282,990 rows. I changed the time out property to 0 also, but that didn't help.
How do I resolve this issue?
I found a hacky solution to it. And that is having a limit at the end of your query. I was facing the same problem with ADO .NET connection to connect to MySQL. Although it doesn't solve the problem. It atleast get the work done.
SSIS: 2208 R2.
MySQL: 5.0
On your OLE DB Destination connection, what "Data access mode" have you selected. If you have selected "Table or view - fast load" (this is the default), then there will be a "Maximum insert commit size" specified. You can try one of two things: 1) change the commit size to a larger number; or 2) try the other data access mode "Table or vew". Since you're getting a timeout error, I suspect that option 1 may not help (since you're already getting a timeout with a smaller value), so try option 2. Although that will likely result in slower performance, it may actually complete successfully. (You could then try #Siva's approach and split the output across multiple destinations to improve performance).
(Note: I'm referring to what's available in SQL Server 2008 R2, if you're using previous versions, it may be slightly different)
If none of the above work, you could also try to create a new SSIS package from scratch by running the SQL Server Import Wizard (right-click on your database in SQL Server Management Studio and select Tasks/Import Data. Follow the wizard screens and near the end make sure you check the box to Save the SSIS package, and choose a file location to save it to. Typically, the resulting SSIS package will be a functional package (and then you can also make whatever further modifications you like to it).
Does MySQL give you the error or are you using PHP (or another language) to transfer the data and does that timeout? In the case of the latter, in PHP you can set the script timeout to infinite using this:
set_time_limit(0);
Either way, based on the information given, I'm not sure what type of database it is, but typically I would set up a cron script to transfer the data bit by bit in order to keep the load at an acceptable level. Please give more information...
I'm trying to create a completely new database from an existing MySQL database, bringing over both data and schema, but so far the only way I've been able to do this is to first import the MySQL database into MS Access, and then into SQL Server 2005? Crazy right? Surely, there is a way that doesn't involve a tedious, custom time-consuming programming, right (perhaps using SSIS)?
A few additions to my original description above:
Its a pretty good size database (easily a few gigs).
I'm working in an MS environment (asp.net, C#)
I'm under a tight deadline so I'm looking for an automated process that requires little to no effort in the conversion process.
SSIS would be the preferred way via BIDS (VS 2005)
Thanks for all the great input!
I believe that using the phpMyAdmin tool you can script the MySQL database structure and data into a sql script. Then you simply run those two scripts on your SQL Server 2005 database and it should, in most cases, create the database and fill it with data. It's been a couple years since I had to do it myself, but as I recall that was the process I used to transfer a MySQL database to SQL Server in the past. You will probably have to alter the structure script to change some of the data types to their SQL Server equivalents, but the data should load just fine once you've got the data types all sorted.
I think you can use SQLYog to generate some fairly standard SQL which will dump out and recreate your db, with data. You may have to massage its output for SQL Server's dialect of SQL a bit, though...
The responses I received were certainly helpful, but the solution it would seem is to do a mysqldump and then run that script from SSIS, massaging the output as needed; however, AFAIK it is not possible to use VS 2005 BIDS to create an SSIS package that completely transfers a MySQL database to a SQL Server 2005 database (data and schema) using Windows Vista 64. I said AFAIK, but who knows the interwebs have much to reveal :)