I have a package that needs to check if a file exists in a folder and if the file does exist then take a branch that will import the file to SQL Server and execute some stored procedures to process it. If the file does not exist then just end the current run of the package without error. I have all parts working just fine except for the file detection and branching depending on the results. (In other words currently it just runs as if the file is there and does the rest). I know how to use a script task to detect for the file and return an error if not found - I need to know how to make the main package just end without error in that case or go on and do the import and the rest of the processing if the file was found.
You could use a Foreach Loop container in the Control flow tab. Loop through a folder for a given pattern (say *.csv). Set the flat file connection manager to use the filepath obtained from the For each loop container as the connection string.
In this setup, the data flow task within the For each loop container will execute only if a file is found. Otherwise, it will end the process silently without any errors.
Here are few other SO questions where I have provided some examples about looping files using Foreach Loop container.
Creating an Expression for an Object Variable?
How can I load a large flat file into a database table using SSIS?
Hope that gives you an idea.
Related
I have an SSIS package that looks in a folder, loops through each file inside, imports file to SQL Server, executes a SQL task, then needs to delete the file, then loop through next one. The import and looping works without the delete file step (File System Task), but the delete file errors with the message "Failed to lock variable. The variable cannot be found." The variable in question is the variable I created for "Current File". It's being used by the first part of the For Each Loop container to look up the current file and import it successfully. What I think is happening, importing the file locks the variable. Then when it's going to delete the file based on the variable, it can't access the variable because it is locked, so it fails. Any idea how to allow it to import the "Current File" based on the "Current File variable", then delete it based on that same variable, then loop through the rest of files? Taking the delete file control out of the For Each Loop is not an option, because I need to delete each file after it imports - if I do a delete after all import, I might delete files in the directory which got there after I ran the import, so I'd be deleting non-imported files, I think. Thanks for any help!
A workaround to this issue would be to create a new object variable. In your current loop you can populate it with the full path and file name of each file that is imported. Then create a second "For Each Loop Container" control task that is enumerated on that object variable. Then within this new For Each Loop Container you can delete only those files (which were imported earlier) defined in the object variable. This way you do not delete any of the other files in the folder that you do not wish to touch.
The following link may help provide some idea as to how to set up the object variable in a For Each Loop Container and then parse out one file name at a time for deletion. Get List of Files
Let me know if this helps or if you need additional details.
I'm trying to create a package that imports excel files into a database using SSIS.
As the operation has to perform this regularly and the file names follow a convention but are not the same, and equally the sheet/tab names are not always the same, the SSIS package is set up as follows:
Main Container
->
First For Each container (call it FE1)
Obtains filenames (assigns to a variable)
->
Second For Each Container (call it FE2)
Obtains worksheet name and starts the process to import.
What I have done is create a "failure" precedence constraint from FE2 to a file system task process in FE1.
The idea is that the file move is done if the import is unsuccessful for whatever reason.
(once it works I'd like to create a "success" process that moves the file to the archive folder)
The file task process works when there is only one "for each container" (i.e. not nested the way it is now) but it fails when all the processes are in the nested container citing "file in use". I'm assuming this is because the first for each container is locking the file, hence why I moved the file task process to the first for each container and used a precedent control.
Any help and advice much appreciated.
For the benefit of anyone else who may have the same problem:
for love nor money could I get the excel connector to release the files, even when the move file task was outside of the loop.
In the end I recorded the files that were moved into a table in the DB and then executed a second package that contained the move file task and would iterate through the table rows with the list of successfully imported (and failed import) files and moved them to their destination based on fail/success flag.
It was the only way I got to successfully make this happen.
When having to iterate through the worksheet of each excel file (i.e. two excel connections effectively) SSIS would not release the files so I forever received the file in use error and failure.
I created a simple SSIS package to import a flat file (.txt) into a database table. Tested that and it works perfectly. Since I have several files to import, I added a foreach loop to go through all the files, added the variables as recommended in several examples found on the net but now my flat file connection manager returns an error of "A valid file name must be selected." and the package will not run. I have so far been unsuccessful in finding the solution for this issue and would appreciate any suggestions by the SSIS gurus of this forum. Many thanks in advance!
Here is what I have in the way of variables:
SourceFileFolder which is the path to the folder that contains the files
FileName a string containing one of the names of the files I am seeking to import
SourceFilePath which is an expression driven variable that incorporates the previous two variables concatenated together. I can click "Evaluate Expression" and copy and paste it into windows explorer and open the file
ArchivePath which is an expression driven variable that creates the path to archive the file to once it is processed.
As the message says this is related to your connection manager not gathering the connection string. This can be handled using the following:
First of all clear the expression on the SourceFilePath variable.
With your Foreach Loop Container, set it up as follows:
This will use your variable SourceFileFolder as the Folder, you could also just hardcode the folder name C:\ for instance. Also make sure your folder is qualified correctly, I.E. make sure it finishes with a slash C: won't work but C:\ will work.
Next you need to map the fully qualified name to your other variable SourceFilePath
This should now store the full name of the file the loop has found into the SourceFilePath variable. For Instance C:\File.txt, you can now use this as a connection string expression on your file connection manager.
Under the properties of the connection manager make sure the expression is set to ConnectionString and then use the SourceFileName variable.
ALSO MAKE SURE DELAY VALIDATION IS SET TO TRUE
This hopefully should mean you can loop through the files.
I have a situation whereby I want to process files in an SSIS package but only files that are new and only files that match specific filename patterns.
Is it possible to use WMI to achieve this task by somehow looping through the resulset of a WMI query?
The WMI Data Reader task seems to be the closest contender but it can only write its results to a file (rather than to say a database table or in-memory recordset).
Has anyone had success doing this?
If you want to use the WMI Data Reader Task then the easiest solution would be to save the result to a file. Add a Data Flow Task that reads the file and inserts the data into the database.
However, another solution would be something like:
Add a Foreach Loop with an Foreach File Enumerator, you can use an expression for the filename patterns.
Process the files in a Data Flow Task
If you are allowed to move the files then use a File System Task to move the file to a different folder so it won't be processed again.
If you can't move the files then you need some other way to determine if the file is already processed. If you only need to watch for new files and not modified ones then you could keep a record of which file has been processed in the database, or add a script task to check the modified date of the file and compare it to the last processed date from the database.
In the figure below, why is the Foreach Loop Container failing despite the fail path (of the DFT that failed) being handled correctly?
How can I get the loop to continue after handling fail path?
If it helps to know what's going on in the package, here's the gist:
We have a requirement where data from Excel files must be loaded into
a DB. The package we have splits each Excel file into constituent CSV
files (one CSV per sheet), and loads the CSVs into the DB. It is
possible that some of the sheets have issues (missing columns, data
type mismatch etc), and such erroneous CSVs are captured by the fail
path of the DFT. Ideally the package must resume processing the rest
of the CSVs, and the rest of the Excel files, and exit successfully.
Do you have any OnError EventHandlers defined for that Data Flow Task? If yes, you could as well set the System Variable, Propogate (type Boolean), for that Error Handler scope to 'False'.
Also please go through Gracefully Handing Task Error in SSIS Package
There is a property on every SSIS component that is called MaximumErrorCount which defines the number of errors that this particular component can accept before failing the whole package.
You have to increase this value for each and every component that you want to continue executing before failing.