Parallel execution in ssis - ssis

Question

I have developed one master package(Main.dtsx) and 3 child packages (Processor.dtsx).Note: Code is same for all child packages that picks up files from source location and process. To optimize the performance, I want that all these 3 child packages should run simultaneously on 10000 files in such a way that first child will pick 1st file and start execution , at the same time second will pick up 2nd file and so on. Please share the code if you have. I tried with 'MaxConcurrentExecutables' option but in that case all components access same file which is not expected.

Answer 1

This cannot be done with a Foreach Loop, but you can accomplish the task with a Script task:
Add 3 string variables to hold the file names (i.e. File1, File2, File3)
Pass the variables from the master package to each child package.
In each child package, configure an expression in the file connection manager to use the parameter as a connection string
At the end of each package, make sure that the file is moved from the source folder or renamed in such a way that it will be ignored in subsequent loops.
Set up a For loop that will end when all the files have been processed. You can add a boolean variable to the package like "ProcessingIsAllDone" and then set this in the script task.
At the top of the For loop add a script task and connect the execute package tasks with precedent constraints.
Use the script below to set the variables
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;
namespace ST_e4ccd9cfaa4847ff86ec88c215c1961c
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
DirectoryInfo sourceDirectory = new DirectoryInfo(#"c:\temp");
int loops = 3;
foreach (FileInfo sourceFile in sourceDirectory.GetFiles("*.txt"))
{
if (loops == 0)
{
break;
}
string variableName = String.Format("File{0}", loops);
Dts.Variables[variableName].Value = sourceFile.FullName;
loops--;
}
if (sourceDirectory.GetFiles("*.txt").Length <= 3)
{
Dts.Variables["ProcessingIsAllDone"].Value = true;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
#region ScriptResults declaration
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}