I have an SSIS package that creates a text flat file from data in a database table. Everything works perfect except that I need to capture the dynamic filename for use in another process. I've searched everywhere but haven't found anything close to what I need other than using a ForEach Loop to loop through the directory the file will be stored in. I can't do that because there's too many things that could go wrong. I'm currently creating the dynamic filename through variables and it contains a datetime stamp.
Is there a way that I can capture the file name when the file is created in the data flow task so I can use it in another process within the control flow task?
Thank you in advance!
John
Related
One folder is having thousand files and each files will be loaded to different sql server tables. How to design a SSIS package to do the task ?
For ex:
File name: Location_12345.xlsx will be loaded into Location table
Employee_1233.txt will be loaded into Employee table
Department_123456.csv will be loaded into Department table
The answer is yes you can. Loop through the files in the folder and get whatever you need. A simple google search would give you everything you need. For example here
You will need a package for each flow. You defined a minimum of 3 in your question.
For example, if all Locations follow the same flow then:
Add a foreach and choose files to loop through.
Define the folder
Define the criteria (Location*.xlsx)
Set the full file path to a variable
Add an excel connection
Make an expression for that variable
Design your data flow.
Delay validation on Excel Source before running.
This is a sample. You will have to do this for each file type.
I'm writing a (what I thought to be a) simple SSIS package to import data from a CSV file into a SQL table.
On the Control Flow task I have a Data Flow Task. In that Data Flow Task I have
a Flat File Source "step",
followed by a Data Conversion "step",
followed by a OLE DB destination "step".
What I want to do is to move the source CSV file to a "Completed" folder or to a "Failed" folder based on the results of the Data Flow Task.
I see that I can't add a File System step inside the Data Flow Task, but I have to do it in the Control Flow tab.
My question is how do I do a simple thing like assign a value to a variable (I saw how to create variable and assign them a value at the bottom pane of Data Tools (2012)) depending of if the "step" succeeds or fails?
Thanks!
(You can tell by my question that I'm an SSIS rookie - and don't assume I can write a C# script, please)
I have used VB or C# scripts to accomplish this myself. Since you do not want to use scripts I would recommend using a different path for the project to flow. Have your success path lead to moving the file to completed and failure path lead to moving the file to failed. This keeps it simple and accomplishes what you are looking for.
I have to create a job which imports data from .csv file into database table everyday. I have created the job and it works fine as long as the filename is same.
The file that comes in everyday has a different name, so i am trying to set up dynamic flat file connection. As i have only one file to load everyday i am not trying to use For Each Loop Container and also, i am not good at script task.
I was trying to see if there is any other way to achieve this using SSIS.
I have created a variable for my path "C:\Daily Files\" as #[User::MyFilePath]. I am then using this variable in the ConnectionString Expression property of the Flat File Connection Manager.
But it does not work. I get an error saying the Cannot open the datafile.
Can someone tell me what am i missing here?
If you need to create a connection manager to a file that changes name everyday, you will have to write some kind of expression to do so. On the other hand, the easier way out is to use the For Each Loop container as it does not care what the name of the file is as long as you give it *.csv in the qualifying field.
I have a situation whereby I want to process files in an SSIS package but only files that are new and only files that match specific filename patterns.
Is it possible to use WMI to achieve this task by somehow looping through the resulset of a WMI query?
The WMI Data Reader task seems to be the closest contender but it can only write its results to a file (rather than to say a database table or in-memory recordset).
Has anyone had success doing this?
If you want to use the WMI Data Reader Task then the easiest solution would be to save the result to a file. Add a Data Flow Task that reads the file and inserts the data into the database.
However, another solution would be something like:
Add a Foreach Loop with an Foreach File Enumerator, you can use an expression for the filename patterns.
Process the files in a Data Flow Task
If you are allowed to move the files then use a File System Task to move the file to a different folder so it won't be processed again.
If you can't move the files then you need some other way to determine if the file is already processed. If you only need to watch for new files and not modified ones then you could keep a record of which file has been processed in the database, or add a script task to check the modified date of the file and compare it to the last processed date from the database.
I am currently using SSIS 2008 and am fairly new to it. I have a programming background with some Java, VBA, and VB.NET.
I have a connection to a csv file that contains a list of URLs.
There about a thousand rows in the file and with each row, I want to add the URL to a package variable that will be used to see if the most current link has already been downloaded and updated or not.
I've set up a Foreach Loop Container that is intended to loop through each row of of the csv file.
However, I cannot figure out how to "look at" each row. Once I can do that I know it will be no problem to assign the URL to the variable but I am stuck mid-way. Does anyone have any suggestions?
You want to do something to each row from a given source. That's usually a data flow type of activity. Drop a Data Flow Task onto your Control Flow. Inside that data flow, add a Flat File Source. In the Flat file connection manager, click New and fill out the details for your file. I assume it's just one data element (url) per line. Click OKs and then you should have a working data source.
Great, now all you need to do is that "something" to the data coming in which in your case is "see if the most current link has already been downloaded and updated or not." I'm not sure exactly what that translates to but whatever you attach (look up task, script task, etc) to the output from the Flat File Source will perform that operation for every row flowing through it.