How can I check how long it lt took to extract data and load data in SSIS.
I am using Attunity connector now for extracting Data from Oracle and then using OLE DB destination to load it in SQL Server. Using Attunity is not making any difference in speed. It took approx 5 minutes to extract and load 5,600,000 rows from oracle to SQL Server when I use attunity connector which is same as OLE DB Source.
In the Progress tab, you can see the time taken for completion of data load.
As others have pointed out, the package duration is included in the GUI and you can also write the start/end times to a logging table within the package to capture the execution time. If you're looking for more detail on specific components, enable logging, choose the PipelineComponent event, and look for this event on the tasks that you're looking to monitor. The PrimeOutput step, which is when the data is sent downstream, will be of the most interest.
Related
I am new to SSIS is there is any component to load data from MYSQL to SQL server using SSIS. Currently am loading data using ODBC connection it is really slow and it around the speed of 30000 rows/Minute. Is there any way to make the load run fast.
Thanks in Advance...
You can install the .NET Connector for MySQL: http://dev.mysql.com/downloads/connector/net/
Then you can create a script task to act as a data source, import MySql.Data.MySqlClient, and query MySQL directly in C#. The data will then enter your Data Flow and you can map it to a SQL Server destination the same as normal.
I find that when using the SSIS connection manager with .Net Providers I get malformed SQL errors, but this way you write all the SQL yourself.
To improve the performance, we can add Conditional Split Transformation build some parallelism into the common data flow that directly load data from ODBC Source to OLE DB Destination.
For more information about speeding up SSIS Bulk Inserts into SQL Server, please see the following blog:
http://henkvandervalk.com/speeding-up-ssis-bulk-inserts-into-sql-server
In DataFlowTask property, Increase buffer size and no of row commit
We've a MySQL 5.1 server and we are trying to import data from SAP Data Services from another server. It works but it takes 3h!
We tried to setup an Oracle XE server to test the same operation and it takes only 10 minutes.
I think the problem is that with Oracle the "Rows per commit" setting (1000) works ok, but with MySQL every row is commited. Is it possible? How can we solve it?
Screenshot of the exporting settings
SAP BusinessObjects Data Services (BODS) allows you to control the number of rows per commit, if data flow ends with a data store in your MySQL server
BODS includes an optimisation PDF guide as part of the help documentation - check that for other recommendations.
We are in the process of moving our backend from ms sql server to mysql. Actually we currently use a couple mysql servers, but mostly ms sql server. I mention this because we are not totally new to mysql. Each day we do a lot of ETL to keep our backend in sync with a legacy system. We move a lot of data and working with sql server has been so much easier than working with mysql for ETL. I know SSIS is MS, but still it has been a headache.
We are using sql server 2012 and BIDS 2010. It has been a struggle to move mysql data at the same rate as ms sql data. We are mainly dealing with innodb tables in mysql. To summarize I have been using the mysql ODBC connector and the ODBC destination in SSIS. The first step is to turn autocommit off on imports. Even with that setting off I can see in package execution that the data source ends up waiting on the destination. It gets about 40,000 rows ahead and waits.
Next I export the data to a text file and then import using a sql task and the INFILE command. This gives pretty good performance, but at the expense of more moving parts. I've had a couple issues with this approach, but it does work and perform well.
Lastly I tried a 3rd party SSIS component from Devart. It creates custom mysql source and destination components. The performance isn't as good as INFILE, but it's not bad and it makes the package simple like when dealing with sql server... a data source and a data destination. No messing with auto commits, exports, INFILE, etc. However I can't use the connections to do other tasks like truncate tables and stuff. So I still have my ODBC connection to do those tasks. I'm going to ask Devart about this.
Right now it looks like Devart is going to be a nice balance. If I absolutely need the performance I have the INFILE method.
I also tried the mysql net connector and could not get that to work at all. I'm running on Windows 7 64bit with Sql Server 2012 64bit. Basically everything I need in BIDS runs in 32bit so I'm guessing this part of issues.
My question is what are others doing when it comes to moving mysql data with SSIS? It has been such a hassle. It would be nice to get some input on what others are doing. What methods are you using? Are you using 3rd party components? Is there a better/dedicated place to discuss SSIS and mysql?
We have developed a simple web app which allows a user to upload an excel file. Our system picks the excel file and triggers an SSIS simple task to read an incoming excel file with 6 columns and push the data in the database.
Our concern is what will happen when mutiple users try to upload files simultaneously? Can SSIS serve multiple requests at the same time (which it should)?
What is the upper limit ie how many requests can it perform simultaneously?
What advice and suggestions you can give us in order for us have our server configured so that we can serve maximum requests?
same SSIS package can run more that once at the same time. The concerns you must have are the same concerns as if you were running 2 insert statments at the same time. SSIS is only the way you are using to insert the data. What happens when the data reaches the SQL Server engine is regardless of how it got there.
Just rememebr that SSIS uses bulk inserts
I am trying to transfer 67,714,854 rows from MySQL to SQL Server using SSIS. The package times out after transferring 14,282,990 rows. I changed the time out property to 0 also, but that didn't help.
How do I resolve this issue?
I found a hacky solution to it. And that is having a limit at the end of your query. I was facing the same problem with ADO .NET connection to connect to MySQL. Although it doesn't solve the problem. It atleast get the work done.
SSIS: 2208 R2.
MySQL: 5.0
On your OLE DB Destination connection, what "Data access mode" have you selected. If you have selected "Table or view - fast load" (this is the default), then there will be a "Maximum insert commit size" specified. You can try one of two things: 1) change the commit size to a larger number; or 2) try the other data access mode "Table or vew". Since you're getting a timeout error, I suspect that option 1 may not help (since you're already getting a timeout with a smaller value), so try option 2. Although that will likely result in slower performance, it may actually complete successfully. (You could then try #Siva's approach and split the output across multiple destinations to improve performance).
(Note: I'm referring to what's available in SQL Server 2008 R2, if you're using previous versions, it may be slightly different)
If none of the above work, you could also try to create a new SSIS package from scratch by running the SQL Server Import Wizard (right-click on your database in SQL Server Management Studio and select Tasks/Import Data. Follow the wizard screens and near the end make sure you check the box to Save the SSIS package, and choose a file location to save it to. Typically, the resulting SSIS package will be a functional package (and then you can also make whatever further modifications you like to it).
Does MySQL give you the error or are you using PHP (or another language) to transfer the data and does that timeout? In the case of the latter, in PHP you can set the script timeout to infinite using this:
set_time_limit(0);
Either way, based on the information given, I'm not sure what type of database it is, but typically I would set up a cron script to transfer the data bit by bit in order to keep the load at an acceptable level. Please give more information...