Loading data from multiple flatfiles with a derived column to a database - ssis

Question

How can i load data from different file sources and the destination table should have a derived column as File_Name and the file name should be displayed for the rows from which file is the data loading into the sql table.
For example:
file1.txt contains
emp_id emp_name
1 abc
file2.txt contains
emp_id emp_name
2 adc
output table should contain
emp_id emp_name file_name
1 abc file1
2 adc file2

Answer 1

I have written several SSIS packages doing exactly what you seek. Assuming you're applying a foreach loop with a "Foreach File Enumerator" (reading from each file with a specific extension in a folder) you can set up an object variable under Collection that will function as an array storing all of your file names (including the complete path) and then under Variable Mappings set up a second Variable of type string (with Index = 0) that will temporarily store the name of the file (again includes the complete path). Call it FileWithFullPath.
There are steps that now need to be applied because you need to send the file name to a database field and the file name must not include the full path or extension (c:\documents\file1.txt becomes file1). To do this you'll need to add to your foreach loop the following in the order defined.
Add a Script Task. Under Script for ReadOnlyVariables, add the FileWithFullPath variable. Under ReadWriteVariables, create a new variable called FileNameOnly. Here we will be reading in the filename that currently includes the full path and then returning the file name excluding the path and extension. Select Edit Script and here you will apply the following C# code to make this happen.
Add "using System.IO;" to region Namespaces above.
Within the braces under public void Main(), add the following:
;
string PathandFileName = Dts.Variables["User::FileWithFullPath"].Value.ToString();
string FileNameWithExtension = Path.GetFileName(PathandFileName); //trim path
string FileNameOnly = Path.GetFileNameWithoutExtension(FileNameWithExtension); //trim extension
Assign final string value to new variable.
Dts.Variables["User::FileNameOnly"].Value = FileNameOnly;
Save and build the code and then exit out.
Finally, under the Control Flow tab add a Data Flow Task into the foreach loop. Select the Data Flow tab, add your source and destination, but in between add a Derived Column task. Here you will create a new column called FileName and drag into its expression the new variable you just populated in the C# code called User::FileNameOnly. This you will then map to the Filename column in your destination database table along with the other columns being read from that file.
Give this a try and let me know if you have any questions.
Thanks.