I need to store the duration of running SSIS packages until execution into SQL database. How can I calculate it.
If you are using SQL 2012+, you are definitely urged to use SSIS Catalogs (SSISDB). Main reason for that - new extensive package logging execution system. View catalog.executions contains necessary for your task data.
If you are on SQL 2008 or run packages out of SSIS Catalogs - then you have to craft something on your side. You might parse results output of DTExec - it always reports runtime duration.
Related
1)
I have a SSIS package and am using package parameter to configure the connections to point to dev. db . Now if i use the environment variables to pass a different connection value pointing to test, after i deploy to the SSIS catalog and further schedule it using the SQL Agent Job.
What connection info. will be taken at runtime , when the job is scheduled to run. Will it be dev or a test conn??
2)
I have a SSIS package and am using project parameter to configure the connections to point to dev. db . Now if i use the environment variables to pass a different connection value pointing to test, after i deploy to the SSIS catalog and further schedule it using the SQL Agent Job.
What connection info. will be taken at runtime , when the job is scheduled to run. Will it be dev or a test conn??
Answer on the first question depends on whether you are using SSIS Package Configurations or not. Possibly not, since you are talking about environment variables, but if you still do and use it together with Environment Variables - see Microsoft article on that, and note this behavior changes with SSIS 2008 version.
If you are using SSIS Catalog and Package/Project variables, then mind this simple rule - more specific wins. In your case, the following precedence will take place, going to the next value if former is missing:
Value specified at package start, either with execution or dtexec parameters.
Value mapped with Environment Variable.
Vales specified at Package design time.
Here is another Microsoft article on package parameter value mapping.
I am new to SSIS is there is any component to load data from MYSQL to SQL server using SSIS. Currently am loading data using ODBC connection it is really slow and it around the speed of 30000 rows/Minute. Is there any way to make the load run fast.
Thanks in Advance...
You can install the .NET Connector for MySQL: http://dev.mysql.com/downloads/connector/net/
Then you can create a script task to act as a data source, import MySql.Data.MySqlClient, and query MySQL directly in C#. The data will then enter your Data Flow and you can map it to a SQL Server destination the same as normal.
I find that when using the SSIS connection manager with .Net Providers I get malformed SQL errors, but this way you write all the SQL yourself.
To improve the performance, we can add Conditional Split Transformation build some parallelism into the common data flow that directly load data from ODBC Source to OLE DB Destination.
For more information about speeding up SSIS Bulk Inserts into SQL Server, please see the following blog:
http://henkvandervalk.com/speeding-up-ssis-bulk-inserts-into-sql-server
In DataFlowTask property, Increase buffer size and no of row commit
we are using SSIS packages to process Cubes in SQL Server 2008 R2 Analysis Services. Until now we have been using an Analysis Services Processing Task in the package and have been manually adding all Cube and Dimension objects to the processing queue in that task. This also means we have to adjust the package when we add Dimensions to a Cube or Cubes to the SSAS DB.
But now we need an SSIS package that will process the whole SSAS database selected, so that we can modify the Cube later on, possibly adding Dimensions, without having to modify the package as well.
In SQL Server Management Studio it is possible to right click an SSAS database and select "Process..." but for the corresping SSIS task, I could not find out how to do this.
Is there any way to process a whole SSAS database in an SSIS package?
Thanks in advance,
Christian
I'm confused as to why you can't use the SSIS Analysis Services Processing Task. I believe you have to option to select a database in the processing settings. You can choose then entire database rather than choosing individual cubes or dimensions on that database. Just make sure the Type says database.
I have also used the XMLA answer provided by #Meff and it works fine as well.
You could also use AMO, you would need to include the Microsoft.AnalysisServices reference to a SSIS script task and provide the variable values. This way doesn't lock you to the database Id but is slightly more complex:
string cubeConnectionString = Dts.Variables["User::CubeConnectionString"].Value.ToString();
string databaseName = Dts.Variables["User::DatabaseName"].Value.ToString();
Server server = new Server();
server.Connect(cubeConnectionString);
Database database = server.Databases.FindByName(databaseName);
database.Process(ProcessType.ProcessFull);
server.Disconnect();
Dts.TaskResult = (int)ScriptResults.Success;
When you go to process the whole database, before you click 'OK', you should see a "Script" button in the top-left of the process window. This will generate the processing XMLA to a new window.
Now take that processing XMLA and use it in an "Analysis Services Execute DDL" control-flow component.
Be careful with cube re-deployments as you'll see the XMLA uses the Id not the name of the database.
I need to migrate all the DTS packages created in SQL Server 2000 to SSIS 2012. What are the differences between SQL Server 2000 and SQL Server 2012. Are there any differences in SQL statements like Insert, Update, Delete etc. What are the things I should be aware of to upgrade the DTS packages to SSIS?
You cannot migrate package directly from DTS written in SQL Server 2000 to SSIS in SQL Server 2012. You could do an intermediate conversion from DTS to SSIS 2005 or 2008 and then upgrade them to SSIS 2012 packages, but I would not advise this, as the conversion wizard is not brilliant and you will also loose most of the benefits of SSIS over DTS.
Therefore, I would strongly advise re-writing the packages in SSIS 2012, replicating the functionality of your original DTS packages. I appreciate that depending on the number of packages involved, this may be a big, time consuming task, but it is the best way.
In terms of differences, I have listed a few more notable ones below:
DTS was COM based, and although under the covers SSIS still uses many COM objects, it is wrapped in .NET
SSIS has sequence containers so that objects can be grouped together
SSIS 2008 and 2012 support C# as well as VB.NET
DTS only allowed mapping column names, but SSIS has a rich set of data transformations
ActiveX scripts, if any, in your DTS package have to be thrown away
In SSIS you need to map Unicode and ASCII manually
SSIS supports 64-Bit
There is no difference between SQL Server 2000 and SQL Server 2012 in terms of basic DML, such as INSERT, UPDATE and DELETE, but SQL 2008 onwards also has a MERGE statement which allows UPSERTs. This is not supported properly in SSIS natively, but there are third party UPSERT components, including a free one on CodePlex.com
Amongst the SSIS Data Flow transformations, however, there are components such as a Slowly Changing Dimension (SCD) component and a OLEDB Command component which allows you to update rows. however, both of these transformations poorly performing and there is usually a better way.
We are doing a huge Data Migration Project using SSIS packages. We were insisted on not using stored procedures in SSIS packages. Can you please suggest whether we should be using stored procedures in SSIS packages or not? What are the advantages of using stored procedures?
It is correct that merge statements can easily be used in SSIS and your directive to encapsulate everything in SSIS is not necessary, as SQL processing aggregations faster than SSIS, for example. Further, if you are not deploying to SSISDB or have proper logging wrappers or email alerts, then troubleshooting your ETL is going to be more difficult via the SQL agent than otherwise as the errors are frequently more cryptic - thus the SSISDB and its reports in 2012. SSIS can be extremely powerful, however.
Here is a fairly blatant benchmark that will tell you never to use the out of the box SCD ever in SSIS. Taskfactory however does have a nice deployable which does basically merges behind the scene.
SSIS has more powerful functions than Stored Procedures.
However you can easily use Execute T-SQL Statement tasks in SSIS for existing tasks, and then build out from there.
SSIS is superior at the vast majority of ETL
Below Via Microsoft
Microsoft Integration Services is a platform for building enterprise-level data integration and data transformations solutions. You use Integration Services to solve complex business problems by copying or downloading files, defining business logic, sending e-mail messages in response to events, updating data warehouses, cleaning and mining data, and managing SQL Server objects and data. The packages can work alone or in concert with other packages to address complex business needs. Integration Services can extract and transform data from a wide variety of sources such as XML data files, flat files, and relational data sources, and then load the data into one or more destinations.
Integration Services includes a rich set of built-in tasks and transformations; tools for constructing packages; and the Integration Services service for running and managing packages. You can use the graphical Integration Services tools to create solutions without writing a single line of code; or you can program the extensive Integration Services object model to create packages programmatically and code custom tasks and other package objects.
A stored procedure in SQL Server is a group of one or more Transact-SQL statements or a reference to a Microsoft .NET Framework common runtime language (CLR) method. They can be called from within SSIS just the same as the unencapsulated SQL statement. For more information about it, please see: http://msdn.microsoft.com/en-us/library/ms190782(v=sql.110).aspx