I have a situation whereby I want to process files in an SSIS package but only files that are new and only files that match specific filename patterns.
Is it possible to use WMI to achieve this task by somehow looping through the resulset of a WMI query?
The WMI Data Reader task seems to be the closest contender but it can only write its results to a file (rather than to say a database table or in-memory recordset).
Has anyone had success doing this?
If you want to use the WMI Data Reader Task then the easiest solution would be to save the result to a file. Add a Data Flow Task that reads the file and inserts the data into the database.
However, another solution would be something like:
Add a Foreach Loop with an Foreach File Enumerator, you can use an expression for the filename patterns.
Process the files in a Data Flow Task
If you are allowed to move the files then use a File System Task to move the file to a different folder so it won't be processed again.
If you can't move the files then you need some other way to determine if the file is already processed. If you only need to watch for new files and not modified ones then you could keep a record of which file has been processed in the database, or add a script task to check the modified date of the file and compare it to the last processed date from the database.
Related
I have an SSIS package that creates a text flat file from data in a database table. Everything works perfect except that I need to capture the dynamic filename for use in another process. I've searched everywhere but haven't found anything close to what I need other than using a ForEach Loop to loop through the directory the file will be stored in. I can't do that because there's too many things that could go wrong. I'm currently creating the dynamic filename through variables and it contains a datetime stamp.
Is there a way that I can capture the file name when the file is created in the data flow task so I can use it in another process within the control flow task?
Thank you in advance!
John
I'm writing a (what I thought to be a) simple SSIS package to import data from a CSV file into a SQL table.
On the Control Flow task I have a Data Flow Task. In that Data Flow Task I have
a Flat File Source "step",
followed by a Data Conversion "step",
followed by a OLE DB destination "step".
What I want to do is to move the source CSV file to a "Completed" folder or to a "Failed" folder based on the results of the Data Flow Task.
I see that I can't add a File System step inside the Data Flow Task, but I have to do it in the Control Flow tab.
My question is how do I do a simple thing like assign a value to a variable (I saw how to create variable and assign them a value at the bottom pane of Data Tools (2012)) depending of if the "step" succeeds or fails?
Thanks!
(You can tell by my question that I'm an SSIS rookie - and don't assume I can write a C# script, please)
I have used VB or C# scripts to accomplish this myself. Since you do not want to use scripts I would recommend using a different path for the project to flow. Have your success path lead to moving the file to completed and failure path lead to moving the file to failed. This keeps it simple and accomplishes what you are looking for.
I have to create a job which imports data from .csv file into database table everyday. I have created the job and it works fine as long as the filename is same.
The file that comes in everyday has a different name, so i am trying to set up dynamic flat file connection. As i have only one file to load everyday i am not trying to use For Each Loop Container and also, i am not good at script task.
I was trying to see if there is any other way to achieve this using SSIS.
I have created a variable for my path "C:\Daily Files\" as #[User::MyFilePath]. I am then using this variable in the ConnectionString Expression property of the Flat File Connection Manager.
But it does not work. I get an error saying the Cannot open the datafile.
Can someone tell me what am i missing here?
If you need to create a connection manager to a file that changes name everyday, you will have to write some kind of expression to do so. On the other hand, the easier way out is to use the For Each Loop container as it does not care what the name of the file is as long as you give it *.csv in the qualifying field.
I have a package that needs to check if a file exists in a folder and if the file does exist then take a branch that will import the file to SQL Server and execute some stored procedures to process it. If the file does not exist then just end the current run of the package without error. I have all parts working just fine except for the file detection and branching depending on the results. (In other words currently it just runs as if the file is there and does the rest). I know how to use a script task to detect for the file and return an error if not found - I need to know how to make the main package just end without error in that case or go on and do the import and the rest of the processing if the file was found.
You could use a Foreach Loop container in the Control flow tab. Loop through a folder for a given pattern (say *.csv). Set the flat file connection manager to use the filepath obtained from the For each loop container as the connection string.
In this setup, the data flow task within the For each loop container will execute only if a file is found. Otherwise, it will end the process silently without any errors.
Here are few other SO questions where I have provided some examples about looping files using Foreach Loop container.
Creating an Expression for an Object Variable?
How can I load a large flat file into a database table using SSIS?
Hope that gives you an idea.
I have an SSIS package that needs to lookup two different types of excel files, type A and type B and load the data within to two different staging tables, tableA and tableB. The formats of these excel sheets are different and they match their respective tables.
I have thought of putting typeA.xls and typeB.xls in two different folders for simplicity(folder paths to be configureable). The required excel files will then be put here through some other application or manually.
What I want is to be able to have my dtsx package to scan the folder and pick the latest unprocessed file and load it ignoring others and then postfix the file name with '-loaded' (typeAxxxxxx-loaded.xls). The "-loaded" in the filename is how I plan to differentiate between the already loaded files and the ones yet to be loaded.
I need advice on:
a) How to check that configured folder for the latest file ie. without the '-loaded' in the filename and load it? ..and then after loading it, rename the same file in that configured folder with the '-loaded' postfixed.
b) Is this the best approach to doing this or is there a better way?
Thanks.
You can do it this way, but it might require several complex string expressions.
E.g. create a ForEach loop over .xls files, inside the loop add an empty script task, then a data flow to load this file. Connect them with a precedence constraint and make it conditional: precedence constraint expression will the check if file name does not end with -loaded.xls. You may either do it in script task or purely using SSIS expression on precedence constraint. Finally, add File System Task to rename the file. You may need to build new file name with another expression.
It might be easier to create two folders: Incoming for new unprocessed files, and Loaded for the files you've processed, and just move the .xls to this folder after processing without renaming. This will avoid the first conditional expression (and dummy script task), and simplify the configuration of File System task.
You can get the SQL File watcher Task and add it to your SSIS. I think this is a cleaner way to do what you want.
SQL File Watcher