I am creating an SSIS package that will fetch data from tables in a sql database and store this data in datasets as package variables. A data flow task containing a flat file source will read data from a text file. This source component will feed my custom component which will access the dataset variables to deal with a few lookups and other manipulating. The custom component then creates sqlcommands for relevant insert statements. The issue that I am having, is that I am unsure of how to access these package variables from my custom pipeline component.
For further clarification:
The reason that I am creating a custom component is because the outputs of the source component will only be known at runtime as the flat file source will be changing and so will its columns. I then lookup a "mapping table" in a sql database to determine which flat file columns will map to fields in a table in the database.
I will be creating a script component which creates the data source and maps its outputs to the custom component. The custom component will use lookup data to create the necessary insert statement.
Since your custom component will be inheriting from PipelineComponent, it will have access to the VariableDispenser property and the AcquireConnections and ReleaseConnections methods.
Related
Is it possible to import data from XML file using Bulk Insert option in SSIS package. Without using Data Flow Task?
The main components of an SSIS package are :
Control Flow (Stores containers and Tasks)
Data Flow (Source, Destination, Transformations)
Event Handler (sending of messages, Emails)
Package Explorer (Offers a single view for all in package)
Parameters (User Interaction)
You will need to drag and drop the Bulk Insert task in the Control Flow.
You have two possible options :
Solution 1 :
Read the XML file and import the data via the OPENROWSET function. This function is native to T-SQL and allows us to read data from many different file types through the BULK import feature, which allows the import from lots of file types, like XML. Use a ForEach Enumerator and call your existing stored procedure containing the OPENROWSET function
Solution 2 :
You will need 2 variables with string data types: SourceFolder and CurrentFileName.
SourceFolder defines where the files will come from and is used in either approach.
CurrentFileName is used in the ForEach Loop Container to capture the "current" file.
I am working with a scenario where I have one azure file storage account in which I have different folders, each contains the *.csv file. I want to load each *.csv file in different Azure SQL Database tables dynamically by iterating over the my RootFolder.
The problem I am facing is my *.csv file contains more columns than my destination. When copy activity gets triggered following error is encountered:
You could make your dataset schema as expression. And then use GetMetadataActivity to get schema of each file.
I have a list of file paths in a table. I need to read these file paths copy these files to a drive which is on present on another server without changing the directory structure. Is this possible ?
You could implement this in SSIS in the following way:
Create a Data Flow Task. In the Data Flow Task, add a Data source component to fetch the filenames from the table. Add a Recordset Destination component and connect the two
Create a For Each container. Connect to the Data Flow Task created in step 1. Change the enumerator type to ADO enumerator
In the For Each task, add a Script Task that takes the file name and uses System.IO.File::Copy method to copy the file to its destination.
I am trying to create a SSIS package that will export csv data to an excel file. This package will be called up from C# giving the csv file as input to the package. I am quite new to SSIS and I've been able to produce the expected result with same headers.
I went with the following approach -
Script task - created scripts from the csv headers to create temp table, bulk insert, excel table scripts.
Execute SQL Task - created a temp table in database
Execute SQL Task - Bulk insert csv data into table
Execute SQL Task - Create Excel file
Data Flow Task - OleDB Source to Excel Destination
Execute SQL Task - Drop the temp table created
The challenge I am facing is that my csv may have different headers (both text and number of headers may be different). And I want a single package to serve this purpose.
With the headers being different, the mapping between OleDB Souce to Excel Destination in step 5 above is not working for dynamic headers and is giving unexpected results in the excel output. Is there any way these mappings can be decided at runtime and not at design time.
I don't believe that you can specify the columns or column mappings of a Data Flow at SSIS runtime. You could build the SSIS package on-the-fly, which would allow your C# code to create the column mappings, but the mappings have to be created before the package can run. See Building Packages Programmatically in MSDN for details.
On the other hand, if all you're trying to do is convert a CSV file into an Excel spreadsheet, it would seem logical to me to use the Workbook.SaveAs method of the Office object model.
I have created a transformation component, and basically it accepts data from a source, and will do very transformations before it can save the information in a database.
But, I want to map between the source columns and the database columns that are called up from my transformation component.
I would like it to look like the mapping in Lookup Transformation Editor.
How do I create an editor that will have the same functionality for mapping, so that it will allow a user to draw a line from a source column to a destination column.
I am using Sql Server 2008 and VS2008 for this project.
I realize I will need to create a custom UI for this component, replacing the Advanced Editor that comes up by default.
Use my wrapper for the DtsMappingSurfaceControl: http://toddmcdermid.blogspot.com/2009/07/using-dtsmappingsurfacecontrol-in-your.html