I have an issue regarding a deletion of files based on a selection.
So, I have an SQL table in which there is a column with the file name.
What I have to do is take the file name from there and delete the physical file with that name from a specified path.
What I have tried is to execute an sql task with the selection of full file paths (including file names) which need to be deleted and store them in an object variable and then try to do a ForEach Loop Container with a 'from variable enumerator' to take those results and delete the files using a file system task, but I don't think that's the right approach.
The execution of the package ends in success, but it ends at the ForEach Loop Container without reaching (executing) the file system task, so the files are not being deleted.
Select statement (also added the full path of the file as a column in order for the file system task to know where the physical file is stored) :
select file_full_path from t1
where flag = 'YES'
Do you guys know a solution to this?
What you can do is set the loop to Foreach ADO Enumerator , so your process would be:
1: SQL Task - map file_full_path to an object variable, say 'filesdel'
2: Foreach Loop set to Foreach ADO Enumerator
3: Map the the the for loop items to a string variable, say 'filename':
4: Use the string variable in the File System Task
Related
I'm using the foreach loop container in my SSIS job and I have a folder where we add new files every day containing the name + date.
I need to run the package using the oldest file; after that we load the second one.
Can we change this bulk insert in SSIS jobs?
FI: I used variables to load them (FolderPath and folderName) but they are processed at the same time; I prefer another solution than using script.
Your question is confusing, but I'll try to answer.
If you don't want to use the script component I will suggest...
1.Using a foreach loop container (folder as source) read the file names into a two column table (Filename (full path), datepart from file name)
2.Using another foreach loop container (using ado object as source) select from that table in the desired order to fill ado object.
Load files that match the path
i have to design a SSIS Package which load flatfile whenever it is available
from the path to the table of a database.
how can i implement it on Production Please suggest.
I Use SQL Agent to execute it. will it be possible only using sql or C#, WITHOUT any third
party tool. ?
Thanks
I have created a tutorial below. At last after deploying your package you can create a job in SSMS which will run at a specified time to execute your package.
To complete this objective following tasks are required.
1. Foreach Loop Container: To iterate over a user configured directory for files.
2. Expression Task: To update a variable if file exists.
Steps
1. First goto Solution Explorer double click on Project.params and create a parameter FolderPath of type string, put value like E:\DataDir\SourceFiles.
2. Create user variables FileNameFromFolder (String), FileToSearch (String) assign value that you want to check and create a variable IsFound (Boolean).
3. Drag and drop a Foreach Loop Container from the SSIS Toolbox under Containers section.
4. Double click on the Foreach Loop Container on the left hand side of Foreach Loop Editor click on the Collection. On the right side set Enumerator as Foreach File Enumerator, now for the Expression click on the three dots which will open a Property Expression Editor. Select Directory as property and for expression select #[$Project::FolderPath]. Click OK.
5. Now in Foreach Loop Editor for the value of Files set *.txt, for the value of Retrieve file name select Name only, normally we select Fully Qualified as it returns file name with the complete path. Check the Traverse subfolders if there can be more than one folder inside a folder.
6. On the left select Variable Mappings, on the right side select User::FileNameFromFolder which will automatically get Index as 0. The file names from the FolderPath will be assigned one by one to the FileNameFromFolder variable. Click OK.
7. Drag and drop a Expression Task inside the Foreach Loop Container from the SSIS toolbox present under the section Common.
8. Double click on the Expression Task, in the Expression Builder write following code. Click OK.
#[User::IsFound] = #[User::FileNameFromFolder] == #[User::FileToSearch] ? TRUE : FALSE
9. The Code above compares the file name that we want to check with the file name from the folder, if both are matched it sets IsFound to True (File Exists).
10. Now the value of IsFound can be used with precedence constraint according to need.
I'm trying to get all the file names from a folder directory along with their row counts. (Also file size in bytes if possible) I am using Microsoft Visual Studio 2010 Shell. Here's what I've done so far:
I have created a Foreach Loop Container, set the Enumerator to Foreach File Enumerator and Expressions to a variable to the folder I want to loop over. I left the Files section with *.* and asked to retrieve Name Only. I have changed the Variable Mappings to a New Variable called FullFilePath, Container is Package, Value type is String and Value: is blank.
I then added a Data Flow to the Loop. Added a flat file source, row count, and OLE DB Destination. I changed the Flat file Source properties expression to the same Folder Variable in the Foreach Loop Container Expression. I added the Variable RecordCount to the Row Count function (Int32, value 0). The OLE DB Destination creates a new table with the name OLE DB Destination.
The next step is a Execute SQL Task that does and Insert Into DBO.FileData (FileName,RowCount) Values (?,?). I set 2 parameter mappings - 1) Variable Name from the Foreach Loop Container, FullFilePath and Data Type VarChar, 2) Variable from Row Count, RecordCount and Data Type Long.
I then have another Execute SQL Task that drops the table created by the data flow task. The problem is that with all the these step the Package still does not complete. It actually gets hung up and fails on the pre-execute. It says:
Warning: Access is denied. Error: Cannot open the datafile 'FullFilePath' Error: Flat File Source failed the pre-execute phase and returned error code 0xC020200E.
Anything you see I could be doing wrong? Let me know if pictures would help.
So I figured this out finally. In order to loop over all of the files with varying headers and column counts I decided to change the option in the Flat File Source to unselect "File contains headers." Doing this allowed the all the files to have the same #1 Column, which by default is Column 0(the first column in all of my files is some sort of a numeric field or ID). I was able to map this through row count and insert into a SQL table. Then I was able to finish the Foreach Loop and scribe the file name and row count into another SQL table to record the counts. It is however taking a really really really long time, i.e. it has been running for over 14 hours and it has only counted through 13 files. Granted some files are 250K+ rows but I wouldn't think it would take this long.
I have a Foreach File Enumerator that will read pdf files name from a folder and place the filename into database. However, i wan it to exclude reading filename that has more less than 3 underscore.
AAA_BBB_000004554_060420161906_S1234567H_M.pdf
AAA_BBB_000003345_060420161906_S9876543H_S.pdf
AAA_BBB_000008546_060420161906_S1234123H_V.pdf
AAA_BBB_201604.pdf
etc
AAA_BBB_201604.pdf should be excluded in the loop as the filename only has 2 underscore.
How can i archive that? i did some search and it seems like using expression is the key, but i had no idea how to do it. Kindly help thank you.
This can be done using TOKENCOUNT function in an Expression.
Create 2 variables
FileName of String type
TokenCount of Int32 type
Foreach Loop Container
Use Foreach Loop Container and set the Collection - Foreach File
Enumerator
Specify the folder location where your .pdf files exists
set ".pdf* under Files: Select the radio button Retrieve File Name -
Name only
Map the File Name retrieved
Next, put an Expression task inside the Foreach Loop Container and using the following expression
Next, drop an Execute SQL Task and connect it from Expression task
#[User::TokenCount] = TOKENCOUNT( #[User::FileName] ,"_")
This uses the TOKENCOUNT function - Returns the number of tokens in a string (FileName in your case) that contains tokens separated by the specified delimiters ('_' in your case)
Assign the token count to an int variable - #[User::TokenCount]
In the Precedence constraint Editor, provide the following Constraint Options
Configure the Execute SQL Task
Finally, it should like this
I put script task between Expression task and Execute SQL task for debugging purpose, if you want you my use this
Running the package - let's say you want to load these file names from this folder
Since, we gave the condition in the expression (Token count > 3), after running the package, these file names will be loading in the database
Hope this helps.
I have a complex task that I need to complete. It worked well before since there was only one file but this is now changing. Each file has one long row that is first bulk inserted into a staging table. From here I'm supposed to save the file name into another table and then insert the the broken up parts of the staging table data. This is not the problem. We might have just one file or even multiple files to load at once. What needs to happen is this:
The first SSIS task is a script task that does some checks. The second task prepares the file list.
The staging table is truncated.
The third task is currently a Foreach loop container task that uses the files from the file list and processes it:
File is loaded into table using Bulk Insert task.
The file name needs to be passed as a variable to the next process. This was done with a C# task before but it is now a bit more complex since there could be more than one file and each file name needs to be saved separately.
The last task is a SQL task that executes a stored procedure with the file name as input variable.
My problem is that before it was only one file. This was easy enough. What would the best way be to go about it now?
In Data Flow Task which imports your file create a derrived column. Populate it with system variable value of filename. Load filename into the same table.
Use a Execute SQL task to retrieve distinc list of filenames into a recordset (Object type variable).
Use For Each Loop container to loop through the recordset. Place your code inside the container. Code will recieve filename from the loop as a value of a variable and process the file.
Use Execute SQL task in For Each Loop container to call SP. Pass filename as a parameter like:
Exec sp_MyCode param1, param2, ?
Where ? will pass filename INPUT as a string
EDIT
To make Flat File Connection to pick up the file specified by a variable - use Connection String property of the Flat File Connection
Select FF Connection, right click and select Properties
Click on empty field for Expressions and then click ellipsis that appears. With Expressions you can define every property of the object listed there using variables. Many objects in SSIS can have Expressions specified.
Add an Expression, select Connection String Property and define an expression with absolute path to the file (just to be on a safe side, it can be a UNC path too).
All the above can be accomplished using C# code in the script task itself. You can loop through all the files one by one and for each file :
1. Bulk Copy the data to the staging
2. Insert the filename to the other table
You can modify the logic as per your requirement and desired execution flow.
Add a colunm to your staging table - FileName
Capture the filename in a SSIS Variable (using expressions) then run something like this each loop:
UPDATE StagingTable SET FileName=? WHERE FileName IS NULL
Why are you messing about with C#? From your description it's totally unnecessary.