SSIS Package - Looping through folder to check if files exists - ssis

Can anyone help:
Required: SSIS Package to loop through a folder (containing 100 files) and check whether required files (which are 5/6) are present in that folder.
Does anyone already has code for this - where we are checking for multiple files existence in the destination folder
Regards

Add a Foreach loop container to your Control Flow
Double click it and select Collection. On Enumerator, select Foreach
File Enumerator
Select your folder and the type of file
Select the return type when a file is found. The options are the
whole filename including extension and path, the name and extension
or simply the name of the file found
Select the checkbox if you want the subfolders
Click on the variables option on the left and then new variable or
select an existing variable.
At this point you have each file name on the folder. To prove it, add a script component, double click it, and your variable on the read Only Variable and click on Edit Script. Make your Main like this:
public void Main()
{
System.Windows.Forms.MessageBox.Show(Dts.Variables["FileName"].Value.ToString());
Dts.TaskResult = (int)ScriptResults.Success;
}
now, the comparison you can do several ways. I dont know where do you have the "required files" list, but assuming it is on a database, you can add a data flow task and inside of it send the filename to the DB to do the comparisson.

Related

SSIS file task system to copy multiple input files from different source paths to different destination paths

I have to copy files from source to destination, here the tricky part is, i have many source files located in different shared paths, i need to copy those input files to particular destination folders.
For Example:
Spath1/a.txt --> Dpath1/, spath2/b.txt --> Dpath2/, spath3/c.txt --> Dpath3/, Etc..
Can i use a table to map the input files/source paths and destination folder and use foreach loop to attain the solution?
kindly post your suggestsions.
Welcome to SO Harun
Yes, you can use a table in this case. First add an an object-type variable to the package, let's call it #folders and a second string-type variable, lets call this as #currentSrc and lastly third string-variable we call #currentDst. Then add Execute SQL task to get the list of folders. Set the result set to "Full result set" and add a reference to #folders-variable in the Result Set tab.
Add a For Each loop, set the Enumerator as "Foreach ADO Enumerator", #folders as the ADO object source variable and "Rows in the first table" as Enumeration mode. Add a reference to #currenSrc and #currenDst-variables in the "Variable Mappings" tab.
Now you can use the File System Task to copy files from sources to destinations based on the data you have on a table in the database (make sure to reference #currentSrc and #currentDst variables in the File System task).

File Uploading To table when Available in Path

i have to design a SSIS Package which load flatfile whenever it is available
from the path to the table of a database.
how can i implement it on Production Please suggest.
I Use SQL Agent to execute it. will it be possible only using sql or C#, WITHOUT any third
party tool. ?
Thanks
I have created a tutorial below. At last after deploying your package you can create a job in SSMS which will run at a specified time to execute your package.
To complete this objective following tasks are required.
1. Foreach Loop Container: To iterate over a user configured directory for files.
2. Expression Task: To update a variable if file exists.
Steps
1. First goto Solution Explorer double click on Project.params and create a parameter FolderPath of type string, put value like E:\DataDir\SourceFiles.
2. Create user variables FileNameFromFolder (String), FileToSearch (String) assign value that you want to check and create a variable IsFound (Boolean).
3. Drag and drop a Foreach Loop Container from the SSIS Toolbox under Containers section.
4. Double click on the Foreach Loop Container on the left hand side of Foreach Loop Editor click on the Collection. On the right side set Enumerator as Foreach File Enumerator, now for the Expression click on the three dots which will open a Property Expression Editor. Select Directory as property and for expression select #[$Project::FolderPath]. Click OK.
5. Now in Foreach Loop Editor for the value of Files set *.txt, for the value of Retrieve file name select Name only, normally we select Fully Qualified as it returns file name with the complete path. Check the Traverse subfolders if there can be more than one folder inside a folder.
6. On the left select Variable Mappings, on the right side select User::FileNameFromFolder which will automatically get Index as 0. The file names from the FolderPath will be assigned one by one to the FileNameFromFolder variable. Click OK.
7. Drag and drop a Expression Task inside the Foreach Loop Container from the SSIS toolbox present under the section Common.
8. Double click on the Expression Task, in the Expression Builder write following code. Click OK.
#[User::IsFound] = #[User::FileNameFromFolder] == #[User::FileToSearch] ? TRUE : FALSE
9. The Code above compares the file name that we want to check with the file name from the folder, if both are matched it sets IsFound to True (File Exists).
10. Now the value of IsFound can be used with precedence constraint according to need.

SSIS - How to loop through files in folder and get path+file names and finally execute stored Procedure with parameter as Path + Filename

Any help is much appreciated. I am trying to create an SSIS package to loop through files in the folder and get the Path + filename and finally execute the stored proc with parameter as path+filename. I am not sure how to get the path+filename and insert the into the Stored proc as parameter. I have attached the screenshot for your reference:
Looks like you have the right idea in general and the link #Speedbirt186 provided has some good details but it sounds like there are a couple of nuances that I thought I might point out in regards to flow and variables.
The foreach loop can assign the entire path or the file name or file name & extension to a variable. The latter will be the most help in your case if you don't want to add a script task to split the Filename from the path. If you start by adding 5 variables to your project it will make it a little easier. 1 will be the Source Directory Path, another the Destination (Archive) Directory Path, and then 1 to hold the File Name and Extension assigned by the for each loop. Then 2 additional dynamic variables that simply combine the source directory and file name to get the source full path and the destination with file name to get the destination full path.
Next make sure you set up your database and Excel file connections. In your Excel file connection after setting it up go to Expressions in the properties window and set the "Connection String" property to SourceFullPath. This will tell the connection to change the file path at every iteration of your loop.
Now you just need to setup your loop etc. Add the fore each loop container setting a directory, filter, and choose File Name and Extension.
Now in the expression box on the collection page set the directory property to be that of your Source Directory variable.
The last part of the Fore each loop is to set your variable mappings to store the file name in your variable. so go to that tab choose your file name variable and set index to 0.
At this point you can add your data flow and setup your import just like you would with a normal file (note your default value for your file name parameter should be to an actual file with the structure you will want to import).
After your data flow drop in your Execute SQL task and set it up how you need. here is an example of direct input and you can see an easy way to reference a parameter is simply a question mark (?).
Next in your sql task setup your parameter mapping by adding in the details you need such as:
Now you are on to your file task. Drop your file task and setup as you desire, but choose your destination and source full path variables to tell the task which file to move.
that's it your are done. there is 1 more thing to note though. The way you have your precedence set in the image you posted you show going from your data flow to your sql and to your file task simultaneously. If your stored procedure relies on your file you may want to put it after your sql task. You can always change the constraint options to "completion" if you want to move the file even if your stored proc fails.
What you want to do is to create a variable in your package, call it something like Filename. In the Edit window of the Foreach you can configure that variable to be set (on the Variable Mappings page- set index to 0).
To create a variable, you will need to have the Variables window showing. Use the View menu to show it if it's not currently open.
Then when calling your stored procedure you can pass the then current value of the variable as a parameter.
This link might help: https://www.simple-talk.com/sql/ssis/ssis-basics-introducing-the-foreach-loop-container/

SSIS Foreach loop container dynamic file name and path, exclude file with more than a certain underscore

I have a Foreach File Enumerator that will read pdf files name from a folder and place the filename into database. However, i wan it to exclude reading filename that has more less than 3 underscore.
AAA_BBB_000004554_060420161906_S1234567H_M.pdf
AAA_BBB_000003345_060420161906_S9876543H_S.pdf
AAA_BBB_000008546_060420161906_S1234123H_V.pdf
AAA_BBB_201604.pdf
etc
AAA_BBB_201604.pdf should be excluded in the loop as the filename only has 2 underscore.
How can i archive that? i did some search and it seems like using expression is the key, but i had no idea how to do it. Kindly help thank you.
This can be done using TOKENCOUNT function in an Expression.
Create 2 variables
FileName of String type
TokenCount of Int32 type
Foreach Loop Container
Use Foreach Loop Container and set the Collection - Foreach File
Enumerator
Specify the folder location where your .pdf files exists
set ".pdf* under Files: Select the radio button Retrieve File Name -
Name only
Map the File Name retrieved
Next, put an Expression task inside the Foreach Loop Container and using the following expression
Next, drop an Execute SQL Task and connect it from Expression task
#[User::TokenCount] = TOKENCOUNT( #[User::FileName] ,"_")
This uses the TOKENCOUNT function - Returns the number of tokens in a string (FileName in your case) that contains tokens separated by the specified delimiters ('_' in your case)
Assign the token count to an int variable - #[User::TokenCount]
In the Precedence constraint Editor, provide the following Constraint Options
Configure the Execute SQL Task
Finally, it should like this
I put script task between Expression task and Execute SQL task for debugging purpose, if you want you my use this
Running the package - let's say you want to load these file names from this folder
Since, we gave the condition in the expression (Token count > 3), after running the package, these file names will be loading in the database
Hope this helps.

SSIS Foreach loop container dynamic file name and path , then unzip files

I have a folder having multiple files with the name as
P04_20140326_1234.zip
P04_20130324_58714.zip
P04_20130808_jurhet.zip
P04_20130815_85893.zip
etc
The name is in the format P04_systemdate_*.zip.
I want to pick the folder containing currentdate in the name and unzip it first and load the data from extracted file into the table.eg : file named as A.txt goes into table A, filenamed as B goes into table B and so on...
I guess you have already done the following:
Add a Data Flow
Inside the data flow, add a flat file source, and Ole_DB destination
Configure the flat file source to point to one of your files and connect all the appropriate columns so that data flows from file to database.
If all of this is already working, then let's do the For-Each loop
Create a variable (default to package root level) and call it CsvFileName of type string
Add a ForEach loop (not a For loop)
Change loop type to be a Foreach File Enumerator
Set your folder path and look for *.csv
Under Variable mappings, add the variable "User::CsvFileName" variable, and set the index to 0 - this means that all file names returned from the Foreach loop will show up in the variable.
In the Connection Managers (bottom) right click on the FlatFileSource, and choose properties
Set the DelayValidation to "True"
Click on Expressions, and then click on the ellipsis
Set the ConnectionString property to use the "CsvFileName" variable
Run it. This should load all files. Now, if you just want to restrict it to a date here's what you do:
Create a variable called "FilterDate"
Set the value to whichever date you want to set (20140322, for example)
In the ForEach loop, go to Collections, and then click on Expressions, then click the ellipsis
Set the FileSpec property to be "*" + #[User::FilterDate] + "*.csv"
Now it will only filter the files that you want.