I have to create a job which imports data from .csv file into database table everyday. I have created the job and it works fine as long as the filename is same.
The file that comes in everyday has a different name, so i am trying to set up dynamic flat file connection. As i have only one file to load everyday i am not trying to use For Each Loop Container and also, i am not good at script task.
I was trying to see if there is any other way to achieve this using SSIS.
I have created a variable for my path "C:\Daily Files\" as #[User::MyFilePath]. I am then using this variable in the ConnectionString Expression property of the Flat File Connection Manager.
But it does not work. I get an error saying the Cannot open the datafile.
Can someone tell me what am i missing here?
If you need to create a connection manager to a file that changes name everyday, you will have to write some kind of expression to do so. On the other hand, the easier way out is to use the For Each Loop container as it does not care what the name of the file is as long as you give it *.csv in the qualifying field.
Related
I have an SSIS package that creates a text flat file from data in a database table. Everything works perfect except that I need to capture the dynamic filename for use in another process. I've searched everywhere but haven't found anything close to what I need other than using a ForEach Loop to loop through the directory the file will be stored in. I can't do that because there's too many things that could go wrong. I'm currently creating the dynamic filename through variables and it contains a datetime stamp.
Is there a way that I can capture the file name when the file is created in the data flow task so I can use it in another process within the control flow task?
Thank you in advance!
John
i am getting two text file having same structure with different name every time.
and i need to load the file everytime to table. how can we configure our connection manager
to load data from different file name to a table. Please any one suggest.
You can use a FileName variable in your Flat File Connection Manager's ConnectionString property and dynamically set it to the different name(s) and achieve this.
Here's a detailed article that shows you step-by-step on how to accomplish this.
SSIS - Dynamically set Flat File Connection Manager
I created a simple SSIS package to import a flat file (.txt) into a database table. Tested that and it works perfectly. Since I have several files to import, I added a foreach loop to go through all the files, added the variables as recommended in several examples found on the net but now my flat file connection manager returns an error of "A valid file name must be selected." and the package will not run. I have so far been unsuccessful in finding the solution for this issue and would appreciate any suggestions by the SSIS gurus of this forum. Many thanks in advance!
Here is what I have in the way of variables:
SourceFileFolder which is the path to the folder that contains the files
FileName a string containing one of the names of the files I am seeking to import
SourceFilePath which is an expression driven variable that incorporates the previous two variables concatenated together. I can click "Evaluate Expression" and copy and paste it into windows explorer and open the file
ArchivePath which is an expression driven variable that creates the path to archive the file to once it is processed.
As the message says this is related to your connection manager not gathering the connection string. This can be handled using the following:
First of all clear the expression on the SourceFilePath variable.
With your Foreach Loop Container, set it up as follows:
This will use your variable SourceFileFolder as the Folder, you could also just hardcode the folder name C:\ for instance. Also make sure your folder is qualified correctly, I.E. make sure it finishes with a slash C: won't work but C:\ will work.
Next you need to map the fully qualified name to your other variable SourceFilePath
This should now store the full name of the file the loop has found into the SourceFilePath variable. For Instance C:\File.txt, you can now use this as a connection string expression on your file connection manager.
Under the properties of the connection manager make sure the expression is set to ConnectionString and then use the SourceFileName variable.
ALSO MAKE SURE DELAY VALIDATION IS SET TO TRUE
This hopefully should mean you can loop through the files.
I have a situation whereby I want to process files in an SSIS package but only files that are new and only files that match specific filename patterns.
Is it possible to use WMI to achieve this task by somehow looping through the resulset of a WMI query?
The WMI Data Reader task seems to be the closest contender but it can only write its results to a file (rather than to say a database table or in-memory recordset).
Has anyone had success doing this?
If you want to use the WMI Data Reader Task then the easiest solution would be to save the result to a file. Add a Data Flow Task that reads the file and inserts the data into the database.
However, another solution would be something like:
Add a Foreach Loop with an Foreach File Enumerator, you can use an expression for the filename patterns.
Process the files in a Data Flow Task
If you are allowed to move the files then use a File System Task to move the file to a different folder so it won't be processed again.
If you can't move the files then you need some other way to determine if the file is already processed. If you only need to watch for new files and not modified ones then you could keep a record of which file has been processed in the database, or add a script task to check the modified date of the file and compare it to the last processed date from the database.
I have a package that needs to check if a file exists in a folder and if the file does exist then take a branch that will import the file to SQL Server and execute some stored procedures to process it. If the file does not exist then just end the current run of the package without error. I have all parts working just fine except for the file detection and branching depending on the results. (In other words currently it just runs as if the file is there and does the rest). I know how to use a script task to detect for the file and return an error if not found - I need to know how to make the main package just end without error in that case or go on and do the import and the rest of the processing if the file was found.
You could use a Foreach Loop container in the Control flow tab. Loop through a folder for a given pattern (say *.csv). Set the flat file connection manager to use the filepath obtained from the For each loop container as the connection string.
In this setup, the data flow task within the For each loop container will execute only if a file is found. Otherwise, it will end the process silently without any errors.
Here are few other SO questions where I have provided some examples about looping files using Foreach Loop container.
Creating an Expression for an Object Variable?
How can I load a large flat file into a database table using SSIS?
Hope that gives you an idea.