Is it possible to import data from XML file using Bulk Insert option in SSIS package. Without using Data Flow Task?
The main components of an SSIS package are :
Control Flow (Stores containers and Tasks)
Data Flow (Source, Destination, Transformations)
Event Handler (sending of messages, Emails)
Package Explorer (Offers a single view for all in package)
Parameters (User Interaction)
You will need to drag and drop the Bulk Insert task in the Control Flow.
You have two possible options :
Solution 1 :
Read the XML file and import the data via the OPENROWSET function. This function is native to T-SQL and allows us to read data from many different file types through the BULK import feature, which allows the import from lots of file types, like XML. Use a ForEach Enumerator and call your existing stored procedure containing the OPENROWSET function
Solution 2 :
You will need 2 variables with string data types: SourceFolder and CurrentFileName.
SourceFolder defines where the files will come from and is used in either approach.
CurrentFileName is used in the ForEach Loop Container to capture the "current" file.
Related
I have been told to change my SSIS package from a Stored procedure call to a Web service call. The Stored procedure was picking up a list of files that my SSIS package needs to process. The call returns file name and the complete path to the file, so multiple rows and multiple columns coming back. So I am running a foreach loop on the object variable(the return from the stored procedure) to process the files.
I put the web service call into my package and set the output to a string variable. I then set up a data flow as I was going to try a XML Source with a recordset Destination to put the xml into a object variable to allow my foreach loop to process as it currently stands.
The problem is the XSD definition of the XML. I do not have one.
Is there a way to get the XSD definition or is there a better way to get the results I am after?
At present I have source XML file, from which I am truncating some values using XSLT file. For this I am using “XML task” in control flow.
Then after getting the resultant file manually I am submitting the resultant XML file to XML source and then loading the data into an SQL table.
If there a better approach to do the same in one go ?
Novice to SSIS. So if possible could someone give example or link to any example to accomplish this
The SSIS Package can contain both control flow as well as one or more data flows. you need to include your both XML Task and Data Flow Task (having XML Source for loading data to sql table). Connect these executable and task in control flow using precedence constraints . This SSIS Package will be automated.
Saving Result from XML Task to file or variable.
In XML Task Editor,
Set SaveOperationResult to True
Set OperationResult\DestinationType either to File Connection or
Variable
Define New Variable or File Connection from Destination property or
select existing one.
In XML Source Editor, select "XML data from Variable" or "XML File Connection" and then specify details in the XML location accordingly.
I am trying to create a SSIS package that will export csv data to an excel file. This package will be called up from C# giving the csv file as input to the package. I am quite new to SSIS and I've been able to produce the expected result with same headers.
I went with the following approach -
Script task - created scripts from the csv headers to create temp table, bulk insert, excel table scripts.
Execute SQL Task - created a temp table in database
Execute SQL Task - Bulk insert csv data into table
Execute SQL Task - Create Excel file
Data Flow Task - OleDB Source to Excel Destination
Execute SQL Task - Drop the temp table created
The challenge I am facing is that my csv may have different headers (both text and number of headers may be different). And I want a single package to serve this purpose.
With the headers being different, the mapping between OleDB Souce to Excel Destination in step 5 above is not working for dynamic headers and is giving unexpected results in the excel output. Is there any way these mappings can be decided at runtime and not at design time.
I don't believe that you can specify the columns or column mappings of a Data Flow at SSIS runtime. You could build the SSIS package on-the-fly, which would allow your C# code to create the column mappings, but the mappings have to be created before the package can run. See Building Packages Programmatically in MSDN for details.
On the other hand, if all you're trying to do is convert a CSV file into an Excel spreadsheet, it would seem logical to me to use the Workbook.SaveAs method of the Office object model.
I am creating an SSIS package that will fetch data from tables in a sql database and store this data in datasets as package variables. A data flow task containing a flat file source will read data from a text file. This source component will feed my custom component which will access the dataset variables to deal with a few lookups and other manipulating. The custom component then creates sqlcommands for relevant insert statements. The issue that I am having, is that I am unsure of how to access these package variables from my custom pipeline component.
For further clarification:
The reason that I am creating a custom component is because the outputs of the source component will only be known at runtime as the flat file source will be changing and so will its columns. I then lookup a "mapping table" in a sql database to determine which flat file columns will map to fields in a table in the database.
I will be creating a script component which creates the data source and maps its outputs to the custom component. The custom component will use lookup data to create the necessary insert statement.
Since your custom component will be inheriting from PipelineComponent, it will have access to the VariableDispenser property and the AcquireConnections and ReleaseConnections methods.
I have a package that needs to check if a file exists in a folder and if the file does exist then take a branch that will import the file to SQL Server and execute some stored procedures to process it. If the file does not exist then just end the current run of the package without error. I have all parts working just fine except for the file detection and branching depending on the results. (In other words currently it just runs as if the file is there and does the rest). I know how to use a script task to detect for the file and return an error if not found - I need to know how to make the main package just end without error in that case or go on and do the import and the rest of the processing if the file was found.
You could use a Foreach Loop container in the Control flow tab. Loop through a folder for a given pattern (say *.csv). Set the flat file connection manager to use the filepath obtained from the For each loop container as the connection string.
In this setup, the data flow task within the For each loop container will execute only if a file is found. Otherwise, it will end the process silently without any errors.
Here are few other SO questions where I have provided some examples about looping files using Foreach Loop container.
Creating an Expression for an Object Variable?
How can I load a large flat file into a database table using SSIS?
Hope that gives you an idea.