I have a very simple SSIS package that grabs 1 or more flat files and imports the data into a SQL Server 2012 table, then a separate package archives the file.. The packages work perfectly, but I want it to warn me if there is no file present when it starts.
I know that the Foreach loop generates a warning when there are no files:
Warning: The For Each File enumerator is empty. The For Each File enumerator did not find any files that matched the file pattern, or the specified directory was empty.
I have not been able to find a way to use that warning to trigger an email, as it does not cause the package to fail, and I am not sure if there is a constraint or expression that can trigger a next step based on a warning.
I use the for each loop to populate a variable #[User::FileName] with any filename matching the pattern which is preset.
A notify operator task with the expression #[User::FileName]=="" and completion or success fires no matter what, if there is a file or not.
Is this possible in SSIS?
You can handle warnings by implementing the OnWarning event handler.
Related
I have an SSIS package that looks in a folder, loops through each file inside, imports file to SQL Server, executes a SQL task, then needs to delete the file, then loop through next one. The import and looping works without the delete file step (File System Task), but the delete file errors with the message "Failed to lock variable. The variable cannot be found." The variable in question is the variable I created for "Current File". It's being used by the first part of the For Each Loop container to look up the current file and import it successfully. What I think is happening, importing the file locks the variable. Then when it's going to delete the file based on the variable, it can't access the variable because it is locked, so it fails. Any idea how to allow it to import the "Current File" based on the "Current File variable", then delete it based on that same variable, then loop through the rest of files? Taking the delete file control out of the For Each Loop is not an option, because I need to delete each file after it imports - if I do a delete after all import, I might delete files in the directory which got there after I ran the import, so I'd be deleting non-imported files, I think. Thanks for any help!
A workaround to this issue would be to create a new object variable. In your current loop you can populate it with the full path and file name of each file that is imported. Then create a second "For Each Loop Container" control task that is enumerated on that object variable. Then within this new For Each Loop Container you can delete only those files (which were imported earlier) defined in the object variable. This way you do not delete any of the other files in the folder that you do not wish to touch.
The following link may help provide some idea as to how to set up the object variable in a For Each Loop Container and then parse out one file name at a time for deletion. Get List of Files
Let me know if this helps or if you need additional details.
I have an SSIS package (SQL 2016) that loads files into a database.
At the beginning of the package I have a Foreach Loop container (Foreach File Enumerator). This loop checks to see if there are any files in an error folder. The desired condition is that there are no files in the error folder.
The ETL works well. However, when there are no files in the error folder, the Foreach Loop container generates a warning:
Foreach File - Check Error Folder:Warning: The For Each File
enumerator is empty. The For Each File enumerator did not find any
files that matched the file pattern, or the specified directory was
empty.
Since this is the desired situation (that there are no files) and since my control flow handles the situation either way, is there a way to suppress this warning?
The reason for wanting to suppress the warning is because the warning count on the package is always 1. Sometimes, however, SSIS warnings are important (such as when fields get out sync). I'd prefer not to have packages that always have warnings since they could mask other, genuine, issues.
It sounds like a small thing, so I thought for sure there'd be a way, but I haven't found it. I tried setting an OnWarning event handler on the Foreach loop and setting Propagate to False. But the warning still gets counted as a warning when the package runs.
I think the best way to solve this small issue is to write a very small script task. Just pass input variable with the path to a folder into the script task, check files count and return output variable back and then use the precedence constraint with an expression
Dts.Variables["User::GoFurther"].Value = Directory.GetFiles(Dts.Variables["User::Path"].Value.ToString()).Any();
I have an SSIS package with several data flow tasks. Each one imports a flat file into a table in my DB. I have created a connection manager for each underlying flat file. The package works just fine if all of the files exist. However, even if one of the files is missing, the entire package fails. I don't want this behavior. For whatever files that exist, I want my package to import them. For those that don't exist, I want SSIS to simply ignore them. At least one of the files will always exist. How do I achieve this behavior? I have seen some solutions that involve either scripts or file control tasks, but I'm not sure which is appropriate for my situation.
my solution is
1. make a Script Task for checking the path file:
SSIS Script task to check if file exists in folder or not
2. ValidateExternalMetadata set to False in the source properties
3. link the Script Task with next step if skip and create a Constrain and Variables connection with if the file exist
In the figure below, why is the Foreach Loop Container failing despite the fail path (of the DFT that failed) being handled correctly?
How can I get the loop to continue after handling fail path?
If it helps to know what's going on in the package, here's the gist:
We have a requirement where data from Excel files must be loaded into
a DB. The package we have splits each Excel file into constituent CSV
files (one CSV per sheet), and loads the CSVs into the DB. It is
possible that some of the sheets have issues (missing columns, data
type mismatch etc), and such erroneous CSVs are captured by the fail
path of the DFT. Ideally the package must resume processing the rest
of the CSVs, and the rest of the Excel files, and exit successfully.
Do you have any OnError EventHandlers defined for that Data Flow Task? If yes, you could as well set the System Variable, Propogate (type Boolean), for that Error Handler scope to 'False'.
Also please go through Gracefully Handing Task Error in SSIS Package
There is a property on every SSIS component that is called MaximumErrorCount which defines the number of errors that this particular component can accept before failing the whole package.
You have to increase this value for each and every component that you want to continue executing before failing.
I have a package that needs to check if a file exists in a folder and if the file does exist then take a branch that will import the file to SQL Server and execute some stored procedures to process it. If the file does not exist then just end the current run of the package without error. I have all parts working just fine except for the file detection and branching depending on the results. (In other words currently it just runs as if the file is there and does the rest). I know how to use a script task to detect for the file and return an error if not found - I need to know how to make the main package just end without error in that case or go on and do the import and the rest of the processing if the file was found.
You could use a Foreach Loop container in the Control flow tab. Loop through a folder for a given pattern (say *.csv). Set the flat file connection manager to use the filepath obtained from the For each loop container as the connection string.
In this setup, the data flow task within the For each loop container will execute only if a file is found. Otherwise, it will end the process silently without any errors.
Here are few other SO questions where I have provided some examples about looping files using Foreach Loop container.
Creating an Expression for an Object Variable?
How can I load a large flat file into a database table using SSIS?
Hope that gives you an idea.