I have a parent pkg that calls a few child pkgs. For each child pkg I have a sql agent job that will override some Conenctions values as in dtexec, where you can use the handy /Conn[ection] to make the pkg configuring in a different way simply pointing to a diff SQL SSIS Conf table (common pattern). The problem is that Execute Package Task (called by the parent) does not have any option rather than calling the child pkg itself (I cannot call Execute Package Task passing smth like /Conn[ection] as I can do with dtexec) so a natural coice would be using Execute Process Task to call dtexec on Child pkg with a appropriate /Conn[ection] setup. Based on your experience are there any drawbacks/issues to consider when using Execute Process Task DTEXEC rather than the Execute Package Task or they are the same thing at the end...?
Mario
The way that I've seen this handled is to create a variable to hold the folder path and variables to hold the package names for each child package. These should be added to the Parent Package.
Then in your connection manager for the child package, you can write an expression to set the connection string dynamically. Right click and select properties for the connection in connection manager, then concatenate these two variables.
Expression Code:
#[User::sPkgFolder]+ #[User::sPkgFilename]
In addition you can set up a XML configuration file and set the variables via the XML file so that as you move the packages from environment to environment you don't have to manually change the code base, but only need to change the value of the path in your configuration file. You can also set up as many variables as need to hold all your child packages.
As already exaplined by SWilliams you can send parent package variables down to a child package using the execute package task, then in the child pacakge you can use this variable in an expression to set the connection. This is far less convoluted than using SQL Agent, configurations etc. and has the added bonus that if you run it in parallel it won't get confused.
Related
I have scenario in SSIS. I have a package which is simple data movement from flatfile to database.
I have a specific location and I want to execute that package when file comes on the folder.
Step-By-Step using WMI Event Watcher Task
Create a WMI Connection manager. Use Windows credentials when running locally (you must be an admin to access WMI event info), and enter credentials when running remotely (be sure to encrypt your packages!)
Add a new WMI Event Watcher Task. Configure the WQLQuerySource property with a WQL query to watch a specific folder for files.
WQL is SQL-like but slightly off, here's the example I'm using to watch a folder:
SELECT * FROM __InstanceCreationEvent WITHIN 10
WHERE TargetInstance ISA "CIM_DirectoryContainsFile"
and TargetInstance.GroupComponent= "Win32_Directory.Name=\"c:\\\\WMIFileWatcher\""
Breaking down this query is out of scope, but note the directory name in the filter and the string escaping required to make it work.
Create a For Each Loop and attach it to the WMI Event watcher Task. Set it with a Foreach File Enumerator, and set the folder to the folder you're watching.
In the Variable Mappings tab of the For Each Loop editor, assign the file name to a variable.
Use that variable name to perform actions on the file (for example, assign it to the ConnectionString property of a Flat File connection and use that connection in a Data Flow task) and then archive the file off somewhere else.
In the diagram below, this package will run until a file has been added, process it, and then complete.
To make the package run in perpetuity, wrap those two tasks in a For Loop with the EvalExpression set to true == true.
You can also consider registering object events using PowerShell and kicking off your SSIS package when those events are triggered. This requires a little less continuous overhead of having your package constantly running, but it adds an extra dependency.
The WMI solution is interesting, but the environment / setup requirements are a bit complex for my tastes. I prefer to resolve this using a ForEach Loop Container and a Execute SQL Wait task, both inside a For Loop Container.
I configure the ForEach Loop Container to loop over the files in a directory, pointing it at the expected file name. The only task inside this Container is a Script Task that increments a Files_Found variable - this will only happen when files are found.
Following the ForEach Loop Container is an Execute SQL task to wait between checks, e.g. WAITFOR DELAY '00:05:00' for a 5 minute delay.
Both that ForEach Loop and Execute SQL task are contained in a For Loop, which initializes and tests the Files_Found variable.
This solution is a lot simpler to manage - it doesn't require any particular setup, environment, permissions, stored credentials or knowledge of WMI syntax.
We have a SSIS job called ExportData and it accepts the 'ExportType' Parameter. The ExportType parameter can be 'schedule' or 'unschedule'.
I created the Variable called '#ExportType' and created the SSIS Configuration and expose the variable in the Configuration file and called it 'ScheduleConfig'. I copied that file and change the value to 'unschedule' and called it 'UnscheduleConfig'.
I created the SSIS Job in the SQL Server 2008 and set up 2 steps for both 'Schedule' and 'Unschedule'. I attached the correct config file for each Step. But whichever step I run, it always execute 'Schedule'. I am sure and double checked the Config files and Steps.
How can I run 2 different jobs with 2 different configuration files?
I did try by using the SetValues method and it doesn't work too.
I suggest you not store #ExportType in an SSIS Configuration. You can override the value from the DTEXEC command line by adding a SET switch similar to:
/SET "\Package.Variables[ExportType].Properties[Value]";Schedule
You can use the same PackagePath above to override this variable value in the SQL Agent Integration Services Job Step Type on the Set Values tab.
Hope this helps,
Andy
If I understand your requirements correctly. here is a sample package created in SSIS 2008 R2 and does what you are looking for. The sample uses a single package, which is executed under two different SQL agent job steps and both the steps use a different copies of the same configuration file. The configuration files hold different values for the variable used inside the package.
Created a sample SSIS package named SO_10177578. Within the package, created a package variable named ExportType of data type String. Also, placed the Execute SQL Task on the Control Flow tab. This task will help to identify the value being passed to the variable ExportType.
Added the OLE DB connection to a sample database and named the connection as SQLServer. I chose to use SQL Server authentication for this connection manager.
Within the SQL database, created a table named dbo.ExportData with the following strucutre. Id column is a identity column.
The table doesn't contain any data to begin with. This table will be populated with data when the package is executed.
On the SSIS package, clicked on the SSIS menu --> selected Package Configuration option. On the Package Configurations Organizer dialog, selected the Enable package configurations checkbox and clicked on the Add button. Created a new package configuration of the type XML Configuration file. Selected a path to store the file.
Added the Value property of the variable ExportType and the ConnectionString property of the connection manager SQLServer to the configuration file.
On the Execute SQL Task, selected the Connection to be SQLServer and set the SQLStatement property to INSERT INTO dbo.ExportData (PackageName, ExportTypeValue) VALUES (?, ?)
Configured both the parameters to the appropriate variables available on the package. Now the package is ready for deployment.
On the database server's SQL Agent node, created a new SQL job named Test_ExportData. Owner field information is removed to hide sensitive information.
On the SQL job's Steps page, created two steps named Step_01 and Step_02.
Step 1 is configured as below with the package being stored in the path c:\temp\SO_10177578.dtsx.
Copied the package configuration file created using the package and changed the value for the variable ExportType to Schedule. Named the configuration file as ScheduleConfig.dtsConfig. Screenshot shows only part of the configuration file to hide sensitive connection string information.
In step 1 of the job, referred the newly created package configuration file from the path c:\temp\Test\ScheduleConfig.dtsConfig.
Step 2 is configured as below and it uses the same package stored in the path c:\temp\SO_10177578.dtsx as used by step 1.
Copied the package configuration file created using the package and changed the value for the variable ExportType to Unschedule. Named the configuration file as UnscheduleConfig.dtsConfig. Screenshot shows only part of the configuration file to hide sensitive connection string information.
In step 2 of the job, referred the newly created package configuration file from the path c:\temp\Test\UnscheduleConfig.dtsConfig.
Now, both the steps are configured. Executed the new SQL agent job.
Queried the table ExportData. You can notice that the package was executed twice and each execution in the SQL agent job's steps used the appropriate configuration files specified on the steps.
Hope that helps.
you cant do that. The SSIS wont read the new configuration.
You need to call the package twice, one time with config file A and one time with config file B. On each config file you will have the #ExportType variable set to 'schedule' and 'unschedule'.
You can only set a parameter once. Even if you use DTEXEC with the /Config option for example you CANT overwrite a parameter that is already set
I have a package that needs to check if a file exists in a folder and if the file does exist then take a branch that will import the file to SQL Server and execute some stored procedures to process it. If the file does not exist then just end the current run of the package without error. I have all parts working just fine except for the file detection and branching depending on the results. (In other words currently it just runs as if the file is there and does the rest). I know how to use a script task to detect for the file and return an error if not found - I need to know how to make the main package just end without error in that case or go on and do the import and the rest of the processing if the file was found.
You could use a Foreach Loop container in the Control flow tab. Loop through a folder for a given pattern (say *.csv). Set the flat file connection manager to use the filepath obtained from the For each loop container as the connection string.
In this setup, the data flow task within the For each loop container will execute only if a file is found. Otherwise, it will end the process silently without any errors.
Here are few other SO questions where I have provided some examples about looping files using Foreach Loop container.
Creating an Expression for an Object Variable?
How can I load a large flat file into a database table using SSIS?
Hope that gives you an idea.
Is there anything similar to global variable in SSIS? I have 4 variables (FromAddress, ToAddress,...) which will be used in all packages (32).
So if I can set them only once it will be very easy to use in all packages and will save my time.
Please advise.
SSIS has variables that can be global to a package, but to span multiple packages, I can think of the following options
Passsing Variables
Have Main Package define a variable and pass the value as a parameter to all packages that it calls. Call the variable the same name in all packages for easy identification.
Config File
Use the same SSIS configuration file across packages and store the value in there.
Environment Variable
Use a windows environment variable that is read from other packages
Registry Value
Store in Windows registry and read for each package - make sure you store under a tree that all packages can see otherwise you may run into permissions issues. Eg HKLM
Database Lookup
Store the value a table structure.
You can create local variables in your scripts. Any variable you create in a script is local just to that script. You can also create global variables (via the Variables slideout window) which can be scoped for the entire package, or a subset of the package.
You could use Package Configurations (using either a DB, XML file, environment file or registry setting) to hold these values and each of the 32 packages can reference the same configuration, rather than have to set up the variables in each package,
You can use a configuration database to retrieve values like that across multiple packages.
Using package configuration, you can simply do it. There is a great answer by #RodWall
https://stackoverflow.com/a/35069635/1468295
Is it possible in SQL 2008 (SSIS) to specify multiple file filters in the for each loop control?
Something like HH*.* and U*.*.
That or a cool workaround would be great.
Thanks,
I don't think that it is possible to do multiple file types. The only way I know of is to do *.* and conditional logic.
What about a foreach loop with regex support?
http://microsoft-ssis.blogspot.com/2012/04/custom-ssis-component-foreach-file.html
It is possible to specify multiple file extensions. All you need to do is specify in the "Files:" section of the Foreach Loop Editor SampleFileSpec*.* and that will retrieve any files that start with SampleFileSpec regardless of the file type or other trailing characters. You can also create an expression in the FileSpec of the Foreach Loop Editor.
If you need to process multiple known file schemas then you can add multiple data flows in the Foreach Loop Container and set the enabled flag for the data flow based on a conditional statement.
The only advantage I can see to doing this is that you only have to iterate through a folder once with a For Each Loop Container. I would recommend having multiple Foreach Loop Containers with their own dedicated data flows. This would make it easier to maintain the code in the future.
Does this resolve your issue? What use-case are you trying to solve that wouldn't be handled better by separate Foreach Loop Containers?
Another option would be something that I prefer to call a "subpackage". In your case, the package would contain just one ForEach loop, configured to loop over the FileSpec as configured in a package variable. The package itself would receive the value for the FileSpec variable through Parent Package Configuration.
That way you have a generic package that does what you expect it to do, available to be called from any other package. To process files with two different filters, all that you need to do is call the package twice, each time with a different value for the variable.
If you're not in favor of using Parent Package configs, that can be avoided by calling the package using an Execute Process task that calls dtexec.exe, while the value for the parameter is passed through the Arguments property.