Unmatched Files Processing through ForEach Loop Container - ssis

Question

I have some processed and unprocessed files in my Source Folder and the file names
of all the processed files are stored in a table. How can I match the files names of source folder and table prior to ForEach Loop Container and process only unmatched files.

Answer 1

The solution below is a bit elaborate but it's the best I could think of.
STEP 1: Create 2 Variables, both strings.
a)CurrentFile: This will be used for your Foreach Loop Container collection value
b)ToProcess: This will be used to map the result set an Execute SQL Task explained
below
STEP 2: Add an Execute SQL Task into your Foreach Loop Container.
Configure Parameter Mapping as shown below:
Use the script below as your SQL Statement:
DECLARE #ToProcess VARCHAR(1)
IF NOT EXISTS(SELECT [FileNames] FROM [YourFilesTable] WHERE FileNames = ?)
SET #ToProcess = 'Y'
SELECT #ToProcess AS ToProcess
Set ResultSet to Single Row as shown below:
Configure Result Set as shown below:
On the Execute SQL Task, configure the Precedence Constraint as shown below:
Your Foreach Loop Container should look like below:

Answer 2

Before the Foreach Loop, use a Script Task to store the names of unprocessed files in an SSIS object variable, then iterate through this variable to load the new files as you already are. Create an object variable and add this in the ReadWriteVariables field of the Script Task. If you're using an SSIS variable to hold the folder path of the source files as done below, add this in the ReadOnlyVariables field. The Foreach Loop will need to use the Foreach From Variable Enumerator enumerator type. In the Variable field on the Collection page, add the object variable that is populated in the Script Task. As you're probably already doing, add a string variable at Index 0 of the Variable Mapping pane and set this variable as the expression of the ConnectionString property on the connection manager, assuming this is a flat file connection. If this is excel, change the ExcelFilePath property to use this variable as the expression. The example code and referenced namespaces for the Script Task is below and uses C#.
using System.Linq;
using System.Data.SqlClient;
using System.IO;
using System.Collections.Generic;
using System.Data;
string connString = #"Data Source=YourSQLServer;Initial Catalog=YourDatabase;Integrated Security=SSPI;";
string cmdText = #"SELECT DISTINCT ColumnWithFileNames FROM YourDatabase.YourSchema.YourTable";
string sourceFolder = Dts.Variables["User::SourceFilePath"].Value.ToString();
//create DirectoryInfo object from source folder
DirectoryInfo di = new DirectoryInfo(sourceFolder);
List<string> processedFiles = new List<string>();
List<string> newFiles = new List<string>();
//get names of already processed files stored in tavle
using (SqlConnection conn = new SqlConnection(connString))
{
conn.Open();
//data set name does not need to relate to name of table storing processed files
DataSet ds = new DataSet("ProcessedFiles");
SqlDataAdapter da = new SqlDataAdapter(cmdText, conn);
da.Fill(ds, "ProcessedFiles");
foreach (DataRow dr in ds.Tables["ProcessedFiles"].Rows)
{
processedFiles.Add(dr[0].ToString());
}
}
foreach (FileInfo fi in di.EnumerateFiles())
{
//only add files not already processed
if (!processedFiles.Contains(fi.FullName))
{
newFiles.Add(fi.FullName);
}
}
//populate SSIS object variable with unprocessed files
Dts.Variables["User::ObjVar"].Value = newFiles.ToList();