I have a data flow task that gets the data from a proc, counts the rows and adds the rows to a flat file with some 20 columns each with different output column width,specified in the advanced tab of the flat file connection manager. The flat file destination object overwrites the file every time and the name of the file is created dynamically.
Now what I need to do is add a header and footer row to that existing flat file with only 5 columns, each with their own width. Values for the header and footer are not coming from the data set used in the above data flow task.
I think it will be its on flat file connection object with 5 columns in it. Some column values can be gotten from the variables.
How can I append a row of header and footer to the existing file coming from the data flow task. I am not sure how to go about doing that..
Since the header and footer rows use different data, you can use three Data Flow Tasks (one for the header, current output, and footer). On the Control Flow tab, use Precedence Constraints to link each DFT in the correct order. For the header and footer tasks, add the necessary source components and set the output to a Flat File Destination. Create a string variable that will contain the name of the output file that all the results will be written to and set this for the ConnectionString expression on all the Flat File Connection Managers that are used. This will ensure they all write to the same file. On the first (header) DFT, select the "Overwrite data in the file" option on the Flat File Destination to ensure that a new file will be created. This can also be done by using the Advanced Editor, going to the Component Properties pane, and setting Overwrite to true. On the second (current) and footer DFTs, set the overwrite option to false so that the data will only be appended to the file for these tasks.
Related
I have a SSIS package - which within a FOR LOOP CONTAINER I look in a particular location, for a particular file format and import it into a database.
This is working fine - when I have two files the contents of both files are being imported.
So I have a Variable Mapping under my ForLoop which records the fully qualified name. What I want to do is when I import the file is I am also recording the file path of where it has come from.
I'm unsure in my dataflow task where I would put that ? Under the data flow I have my source file and a destination.
I tried to have a sql task after the data flow that updated the field in the database with the variable (via Parameter Mapping), but that set the field to the same value for everything (the last file path found) which is not what I'm after.
Any advice would be welcome
In your dataflow task, in between your source and destination add a Derived Column transformation. This will add columns to your dataset with a name and value that you specify. If you reference variables in which you are storing the file name for your loop container, the name of the file being accessed will be appended to an additional column in your dataset. Obviously you need to make sure that this column is present in your destination table.
I receive a pipe-delimited flat file each week that has 50 columns. I am trying to use SSIS to take that file, delete the last 3 columns, then insert the remaining data into a new pipe-delimited flat file. At first I thought this would be very simple, but I've got a stubborn flat file connection manager. It keeps reverting back to the inbound file layout with the extra columns, and also keeps going back to a comma delimited file when the outbound file needs to be pipe delimited.
The way I'm "deleting" the unneeded columns is by just removing them from the inbound flat file connection manager, so they aren't listed in the output columns from the flat file source, and they don't show to be on the input columns of the flat file destination.
The file name of both files is dynamic...not sure if that's having something to do with it.
I have delay validation set to true for both, but I'm not sure what else to try. I've also tried deleting all of it and adding back in the connection managers and the files.
Is there some issue with having 2 flat file connection managers, one for the source and one for the destination? Is there a setting I'm missing?
Delete your connection managers and Source/Destination (effectively start over)
Add Flat File Source (FFSRC) with new FF Connection
Set up FFSRC as needed (pipe delimited, headers, etc) - don't delete any rows
After clicking "OK", you're back in the Data Flow. Right click on your Flat File Source, and click "Show Advanced Editor"
Go to "Input and Output Properties" tab, then expand the FFSRC Output/Output Columns. Click a column, then click "Remove Column".
Add your Flat File Destination (FFDST) with a new connection manager, and map the inputs.
Your destination shouldn't have those columns now.
If the Flat File connection seems to be resetting because of dynamic names, look into supplying them as expressions/variables.
To do that, click on the Connection Manager node for your source/destination (not the data flow node), then in Properties, expand Expressions. You will want to make the ConnectionString dynamic via a variable.
What do you mean by
It keeps reverting back to the inbound file layout with the extra columns, and also keeps going back to a comma delimited file when the outbound file needs to be pipe delimited.
Have you tried directly referencing the file, then setup the type of file (ragged right, delimited or fixed width), then applying the expression to the connectionstring property of the connection manager?
I suggest ragged right, I can specify whatever 'column' and width I want to.
Does any one have a tutorial on how to import a fixed width flat file into a database using an SSIS package?
I have a flat file containing columns with varying lengths.
Column name Width
----------- -----
First name 25
Last name 25
Id 9
Date 8
How do I convert a flat file into columns?
Here is a sample package created using SSIS 2008 R2 that explains how to import a flat file into a database table.
Create a fixed-width flat file named Fixed_Width_File.txt with data as shown in the screenshot. The screenshot uses Notepad++ to display the file contents. It has the capability to show the special characters like carriage return and line feed. CR LF denotes the row delimiters Carriage return and Line feed.
In the SQL server database, create a table named dbo.FlatFile using the create script provided under SQL Scripts section.
Create a new SSIS package and add a new OLE DB Connection manager that would connect to the SQL Server database. Let's assume that the OLE DB Connection manager is named as SQLServer.
On the package's control flow tab, place a Data Flow Task.
Double-click on the data flow task and you will be taken to the data flow tab. On the data flow tab, place a Flat File Source. Double-click on the flat file source and the Flat File Source Editor will appear. Click the New button to open the Flat File Connection Manager Editor.
On the General section of the Flat File Source Editor, enter a value in Connection manager name (say Source) and browse to the flat file location and select the file. This example uses the sample file in the path C:\temp\Fixed_Width_File.txt If you have header rows in your file, you can enter a value 1 in the Header rows to skip textbox to skip the header row.
Click on the Columns section. Change the font according to your choice I chose Courier New so I could see more data with less scrolling. Enter the value 69 in the Row width text box. This value is the sum of width of all your columns + 2 for the row delimiter. Once you have set the correct row width, you should see the fixed width file data correctly on the Source data columns section. Now, you have to click at the appropriate locations to determine the column limits. Note the sections 4, 5, 6 and in the below screenshot.
Click on the Advanced section. You will notice 5 columns created for you automatically based on the column limits that we set on the Columns section in the previous step. The fifth column is for row delimiter.
Rename the column names as FirstName, LastName, Id, Date and RowDelimiter
By default, the columns will be set with DataType string [DT_STR]. If we are fairly certain, that a certain column will be of different data type, we can configure it in the Advanced section. We will change Id column to be of data type four-byte signed integer [DT_I4] and Date column to be of data type date [DT_DATE]
Click on the Preview section. The data will be shown as per the column configuration.
Click OK on the Flat file connection manager editor and the flat file connection will be assigned to the Flat File Source in the data flow task.
On the Flat File Source Editor, click on the Columns section. You will notice the columns that were configured in the flat file connection manager. Uncheck the RowDelimiter because we won't need that.
On the data flow task, place an OLE DB Destination. Connect the output from the Flat file source to the OLE DB Destination.
On the OLE DB Destination Editor, select the OLE DB Connection manager named SQLServer and set the Name of the table or the view drop down to [dbo].[FlatFile]
On the OLE DB Destination Editor, click on the Mappings section. Since the column names in the flat file connection manager are same as the columns in the database, the mapping will take place automatically. If the names are different, you have to manually map the columns. Click OK.
Now the package is ready. Execute the package to load the fixed-width flat file data into the database.
If you query the table dbo.FlatFile in the database, you will notice the flat file data imported into the database.
This sample should give you an idea about how to import fixed-width flat file into database. It doesn't explain how to handle error logging but this should get you started and help you discover other SSIS related features when you play with packages.
Hope that helps.
SQL Scripts:
CREATE TABLE [dbo].[FlatFile](
[Id] [int] NOT NULL,
[FirstName] [varchar](25) NOT NULL,
[LastName] [varchar](25) NOT NULL,
[Date] [datetime] NOT NULL
)
In the derived column transformation you can use SUBSTRING() function for each of the column.
Example:
Columns DerivedColumn
FirstName SUBSTRING(Data, startFrom, Length);
Here the FirstName has width 25 so if we consider that from the 0th position then in the derived column you should specify it by giving SUBSTRING(Data, 0, 25);
Similarly for other columns.
Very well explained, Siva! Your tutorial and excellent illustrations point out what Microsoft should have made clear
that the width for a fixed length row has to include the Carriage Return and Line Feed (CR & LF) characters (which I figured out because the preview showed the rows were not lining up correctly)
the all important step of defining an extra column to contain those CR & LF characters, even though they won't be imported. I figured this out, too. I would have benefited by finding your answer before I began.
Without those two things, an attempt to run the import will give this error message:
The data conversion for column "Column x" returned status value 4 and status text "Text was truncated or one or more characters had no match in the target code page.".
I have added in this error text in hopes someone will find this page while searching for the cause of their error. Your turorial is worth finding, even if after the fact!
I need to add an footer in flat file destinatiion as an row count. am importing data from oledb to flat file, i need the count of the records placed as a footer in flat file.
I would solve this similar to this:
ssis package format in excel
But in this case you can:
Create variable of type int (package scope).
Add Row Count Component in your data flow and in Variable Name put you var.
Create script task that will connect to your txt file and add additonal row with rowcount.
Hope it will help.
There are only 3 files that can be created : "File_1", "File_2" and "File_3". The same variable name is used in each instance (User::FileDirectory) and (User::File_name), but because the actual value of the variable changes, a new file is created.However the files are only created if there is data to go into the file. i.e. if there are no records to populate the file, it will not be created at all. When the files are created, the date the file was created should also be added to the filename. eg: File1_22102011.txt
Ok if the above was a little confusing, the following is how it works,
All the files use the same variable, but it is reset before each file is created.
• So it populates a result set in memory with the first sql selection (ID number, First_Name and Main_Name). It sets the file variable to “File_1”. If there are records in the result set, it creates and writes to this filename.
• Then it creates a new result set with the second selection(Contract No). It sets the variable to "File_2". If there are records in this new result set, a new file will be created from the variable(which now has a new value)
• Finally a third result set is created (Contract_no, ExperianNo, Entity_ID_Number, First_Name, Main_Name), and the file variable is set to "File_3". Again if there are records in the result set, then this file will be created and written to.
I have worked on a few methods to achieve this but they all have failed, So little help will be greatly appreciated.
While what you have works, I think it'd be rather painful to maintain.
I would approach it as 3 sequence containers running in parallel. Each container would have a data flow and two file tasks hanging off it based on success of the parent and the value of row count variable. If the row count variable is 0, delete the file. If it's greater than 0, rename it to File_n
As you can see, I have a container for the first file. The data flow creates an output a.txt file. Based on the value of the variable #RowCount1, it will either delete the empty file or rename it to File_1.
Each data flow would look like a source query, a row count transformation and a file destination with a temporary name (a.txt, b.txt, c.txt). As a file is always created, even if it's empty, we will need to delete or rename it afterwards which will be accomplished based on the file operation tasks.
In my opinion, this approach will be cleaner as it will allow you to test and debug each item in a cleaner manner rather than dealing with an in-memory dataset.