I created a simple SSIS package to import a flat file (.txt) into a database table. Tested that and it works perfectly. Since I have several files to import, I added a foreach loop to go through all the files, added the variables as recommended in several examples found on the net but now my flat file connection manager returns an error of "A valid file name must be selected." and the package will not run. I have so far been unsuccessful in finding the solution for this issue and would appreciate any suggestions by the SSIS gurus of this forum. Many thanks in advance!
Here is what I have in the way of variables:
SourceFileFolder which is the path to the folder that contains the files
FileName a string containing one of the names of the files I am seeking to import
SourceFilePath which is an expression driven variable that incorporates the previous two variables concatenated together. I can click "Evaluate Expression" and copy and paste it into windows explorer and open the file
ArchivePath which is an expression driven variable that creates the path to archive the file to once it is processed.
As the message says this is related to your connection manager not gathering the connection string. This can be handled using the following:
First of all clear the expression on the SourceFilePath variable.
With your Foreach Loop Container, set it up as follows:
This will use your variable SourceFileFolder as the Folder, you could also just hardcode the folder name C:\ for instance. Also make sure your folder is qualified correctly, I.E. make sure it finishes with a slash C: won't work but C:\ will work.
Next you need to map the fully qualified name to your other variable SourceFilePath
This should now store the full name of the file the loop has found into the SourceFilePath variable. For Instance C:\File.txt, you can now use this as a connection string expression on your file connection manager.
Under the properties of the connection manager make sure the expression is set to ConnectionString and then use the SourceFileName variable.
ALSO MAKE SURE DELAY VALIDATION IS SET TO TRUE
This hopefully should mean you can loop through the files.
Related
I have an SSIS package that looks in a folder, loops through each file inside, imports file to SQL Server, executes a SQL task, then needs to delete the file, then loop through next one. The import and looping works without the delete file step (File System Task), but the delete file errors with the message "Failed to lock variable. The variable cannot be found." The variable in question is the variable I created for "Current File". It's being used by the first part of the For Each Loop container to look up the current file and import it successfully. What I think is happening, importing the file locks the variable. Then when it's going to delete the file based on the variable, it can't access the variable because it is locked, so it fails. Any idea how to allow it to import the "Current File" based on the "Current File variable", then delete it based on that same variable, then loop through the rest of files? Taking the delete file control out of the For Each Loop is not an option, because I need to delete each file after it imports - if I do a delete after all import, I might delete files in the directory which got there after I ran the import, so I'd be deleting non-imported files, I think. Thanks for any help!
A workaround to this issue would be to create a new object variable. In your current loop you can populate it with the full path and file name of each file that is imported. Then create a second "For Each Loop Container" control task that is enumerated on that object variable. Then within this new For Each Loop Container you can delete only those files (which were imported earlier) defined in the object variable. This way you do not delete any of the other files in the folder that you do not wish to touch.
The following link may help provide some idea as to how to set up the object variable in a For Each Loop Container and then parse out one file name at a time for deletion. Get List of Files
Let me know if this helps or if you need additional details.
I have set up part of my ssis package to check a folder and move the csv file to another folder using a file system task.
Is it possible to rename a file without knowing what the source name will be? as I have been told it will be a new name on a daily bases. can I use a wildcard in a file system task? would I be best renaming before moving what is best practise?
For an unknown file name you can wrap your file system task in a foreach loop
wrap your file system task in a foreach loop and set to file enumerate
set the folder to your source location
set search string to *.csv
set to full file path
map to a variable called fname
use fname as variable in file system task for source
Make sure you delay validation on connection manager
I have to create a job which imports data from .csv file into database table everyday. I have created the job and it works fine as long as the filename is same.
The file that comes in everyday has a different name, so i am trying to set up dynamic flat file connection. As i have only one file to load everyday i am not trying to use For Each Loop Container and also, i am not good at script task.
I was trying to see if there is any other way to achieve this using SSIS.
I have created a variable for my path "C:\Daily Files\" as #[User::MyFilePath]. I am then using this variable in the ConnectionString Expression property of the Flat File Connection Manager.
But it does not work. I get an error saying the Cannot open the datafile.
Can someone tell me what am i missing here?
If you need to create a connection manager to a file that changes name everyday, you will have to write some kind of expression to do so. On the other hand, the easier way out is to use the For Each Loop container as it does not care what the name of the file is as long as you give it *.csv in the qualifying field.
I developed a SSIS package that creates several .txt files. These files are zipped and then the .txt files need to be removed. Using a foreach file enumerator, I loop through all the .txt files for a specific folder. The folder is retrieved from a variable in configuration and looks something like: C:\Folder\
The foreach loop uses: *.txt to gather all .txt files, does not traverse subfolder and uses the full qualified name.
In the Variable Mappings the "FileName" variable gets filled with the 0 index.
Within the foreachloop I use a File system task.
This task removes the .txt files which are generated before, using the FileName variable that is filled in the loop.
On the development machine this runs like a charm. All greens, no problem at all. Now I copy the package and the configuration file to the test environment. A basic version without the file removing was running perfectly fine here. I replaced the package. Nothing big.
Now I run the SQl Server Agent Job and it starts running. I can see all the text files appearing, and disappearing after it created the zipfiles. However, when all files are removed the package results with errors. Namely the error shown above in the title.
I tried looking for the connectionmanager that might have been removed
Looked for connection managers named in the config that don't exist in the package.
No such thing found. Annoying part is that the package is fully functioning, but still results with the error.
EDIT: I noticed that if I run the package using the execute package utility with the dev. config it gives the same errors.
Hopefully someone is able to help me out.
Thanks in advance!
I managed to "fix" the issue. Remove the File System Component responsible for deleting the files. Then add it again and configure it again.
I think this happens if you accidentally change General parameters before changing the Operation parameter. It holds the metadata to irrelevant parameters and upon execution says: "Wait, you defined this parameter but I don't need it, but I'm checking for it anyway, and it's not there!"
It's a bug for sure
I have a package that needs to check if a file exists in a folder and if the file does exist then take a branch that will import the file to SQL Server and execute some stored procedures to process it. If the file does not exist then just end the current run of the package without error. I have all parts working just fine except for the file detection and branching depending on the results. (In other words currently it just runs as if the file is there and does the rest). I know how to use a script task to detect for the file and return an error if not found - I need to know how to make the main package just end without error in that case or go on and do the import and the rest of the processing if the file was found.
You could use a Foreach Loop container in the Control flow tab. Loop through a folder for a given pattern (say *.csv). Set the flat file connection manager to use the filepath obtained from the For each loop container as the connection string.
In this setup, the data flow task within the For each loop container will execute only if a file is found. Otherwise, it will end the process silently without any errors.
Here are few other SO questions where I have provided some examples about looping files using Foreach Loop container.
Creating an Expression for an Object Variable?
How can I load a large flat file into a database table using SSIS?
Hope that gives you an idea.