I have the following use case:
I am trying to move the files in one folder to another folder. If any file is corrupt then whole process should be rolled back and no files should be moved.
For achieving this I am making use of one data flow task and one file system task. Data flow task would check for the integrity of the file and file system task would then move the file. These two tasks are in foreach container.
The transaction property of foreach is set to required and for the two tasks inside it, i am keeping it as supported.
Issue: There are 6 files in one folder which are to be moved. The file # 4 is corrupt. I want the whole task to rollback when system detects the corrupt file. However, this is not happening and files uptil file no 3 get moved.
Screen shot attached.
Related
Using Microsoft Visual Studio Community 2015.
Goal of project
-create "*\temp\email" directory
-start program to extract all emails that include xls attachments to the previously created folder
-use for each loop to cycle through each file in the folder, process, and shift to sql table.
The problem I am running into is caused by either a blank excel document (which is occasionally sent from a remote location) or some of the original xls reports only contain 5 columns instead of 6 that I have mapped now. Is there any way to separate files that include the correct columns from those that do not match?
** as Long as these two problems do not exist I can run the ssis package and everything runs without issue.
Control flow;
File System Task (creates directory --->Execute Process Task (xls extraction)-->ForEach Loop(Data flow Task "email2Sql")
Data Flow;
Excel Source (uses expression ExcelFilePath,#user:filepath) delay validation ==true
(columns are initially set to f1-f6 and are mapped to for ex. a,b,c,d,e,f. The Older files that get mixed in only include a,b,c,d,e.) This is where I want to be able to separate the xls files
Conditional Transformation split (column names are not in row 1, this helps remove "null" values)
Ole Db destination (sql table)
Sorry for the amount of reading, but for the first post I tried to include anything that I thought may be relevant.
There are some tools out there which would allow you to open the excel doc and read it. However, I think the simplest thing to do would be to use SSIS out of the box:
1 - add a file system task after the data flow which reads the file.
2 - Make the precedence constraint from the data flow to the file system task "failure." This will cause that to only fire when the data flow task fails.
3 - set the file task to move the "bad" files to another folder
This will allow you to loop through all the files and move the failed ones. Ultimately, the package will end in failure. If you don't want that behavior you can change the ForceExecutionResult property to be success. However, it might be good to know that there were problems with some files so that they can be addressed.
m
I'm trying to create a package that imports excel files into a database using SSIS.
As the operation has to perform this regularly and the file names follow a convention but are not the same, and equally the sheet/tab names are not always the same, the SSIS package is set up as follows:
Main Container
->
First For Each container (call it FE1)
Obtains filenames (assigns to a variable)
->
Second For Each Container (call it FE2)
Obtains worksheet name and starts the process to import.
What I have done is create a "failure" precedence constraint from FE2 to a file system task process in FE1.
The idea is that the file move is done if the import is unsuccessful for whatever reason.
(once it works I'd like to create a "success" process that moves the file to the archive folder)
The file task process works when there is only one "for each container" (i.e. not nested the way it is now) but it fails when all the processes are in the nested container citing "file in use". I'm assuming this is because the first for each container is locking the file, hence why I moved the file task process to the first for each container and used a precedent control.
Any help and advice much appreciated.
For the benefit of anyone else who may have the same problem:
for love nor money could I get the excel connector to release the files, even when the move file task was outside of the loop.
In the end I recorded the files that were moved into a table in the DB and then executed a second package that contained the move file task and would iterate through the table rows with the list of successfully imported (and failed import) files and moved them to their destination based on fail/success flag.
It was the only way I got to successfully make this happen.
When having to iterate through the worksheet of each excel file (i.e. two excel connections effectively) SSIS would not release the files so I forever received the file in use error and failure.
I have a folder which contains files like:
A_ddmmyyyy, b_ddmmyyyy and c_ddmmyyyy.
I need to read all these files for a date and again all these files for the next date present in the same folder. Also the number of files present in the folder varies (may contain data for three days or of five dates), but the date on the folder remains the same.
Is it possible?
You can create a Foreach Loop Container, inside the container create a data flow task that processes all the files in a specific folder. You create a flat file connection with the needed delimiter, and a variable that has the folder path for your files. And then as a last step to your tasks, create a File System Task to move the file to a processed or completed folder so that the your main folder empties out once the files are processed .
File name doesn't matter, you have to be certain how many kinds of schemas (number of columns, column names, types) of files in the folder. Say you have 3 schemas, then you will need to define 3 type of flat file connections. There are many ways to do the job, the easiest of i can think of is to use powershell to separate files of different schema into different folders, you have to know what file names are mapped to what schema, there may be a pattern or business rule. You then put your powershell in a execute process task to run the script. Then the following is simple, for each folder you create a package, inside is a for each container where you loop through the folder to load each file. Or you can have one package with three for each loop container to do the job.
I have a process that I want to create in SSIS. What I have so far is below.
So the scenario is that a number of different file types are downloaded to a specific location - which has been created as a variable "RootFolder"
Within that RootFolder, is a folder called "Archive"
The first part of the process is to create a folder within the variable of ArchiveFolder, which is basically todays date.
Then in a For Loop Container, it copies all the files in the RootFolder to the Archive Folder. So we have a history of the files received.
The next step that I want to carry out in SSIS is to add a header line at the top of each of those text files. So all the files that still exist in the Root Folder get a header added to the top of the file. Is there something direct in SSIS to do this, or would I need a .bat file to do it and have an Execute Process Task, to call that bat file. Either way I have no idea how to achieve this bit. Your help will be appreciated.
I am new to SSIS. I want to check the error state. If failure, I want to move the file to the error folder. If success, I want to move the file to the success folder.
Example: If number of rows is less than the flat file is update in the DB it should prompt an error and drop the file in to folder and next file should continue updating in the DB.
You could create a For Each Loop that iterates once per file you want to process. By default, this loop's MaximumErrorCount will stop it from iterating after the first failure. Since you want it to still continue on through the other files, set MaximumErrorCount on the loop to 0 (0 = infinity).
Place your data flow inside the loop. Then, attach two File System Tasks to the data flow--one with a precedence constraint of success and the other of failure. If the data flow completes with a success status, the "success" File System Task will run, moving the file to wherever you'd like. If the data flow completes with a failure status, the other File System Task can copy the file to the appropriate failure location.