I need to query three different database and dump them into csv files. Its the same procedure for the three databases. The only difference is the database and the name of the csv file. Can I do this without cutting and pasting? Is there a way to pass parameters to the data flow task?
Thanks!
Your flat file and db connection managers could have the connection string based on a package scoped variable.
Then use a foreach looping container to call your dataflow task. Configure the looping container with a foreach item enumerator and add the appropriate names to the collection.
santiiiii's explanation covers the use case of downloading the data in one package execution. If you need to get the data at different times, then you can use a conditional statement in a variable that will give you different file names and database connections based on the supplied value for the variable. You can then set the value of the variable in the SQL Server Agent Job in the Set Values tab. This can give you more flexibility, but santiiiii's solution is definately best if you want to process all three files at the same time.
Related
I have got around 35 tables whose data need to be migrated from SQL Server to MySQL. I am using SSIS for this project and I have set up a control flow (using Load Multiple Tables) with a Script Task and a Foreach Loop Container that iterates through all the tables in my database. What I now need to do is convert the data type for some of the columns, in some of the tables, to 'Unicode String [DT_WSTR]' before I dump them in my destination tables. Is this something that can be done through SSIS? If so, any pointers or a set of instructions would be great.
Thanks,
Pratik Gandhi
Yes, this is a standard out-of-the-box task for SSIS.
Add a Data Flow Task.
Add a Data conversion component to the task
Add your source and destination servers
Map your columns, converting datatypes where required.
As always, MSDN provides further help.
I am new in PDI (passing from SSIS) and I am having some troubles by handling the variables issue.
I would like to perform this:
From a sql select query I would like to save the result into a variable.
For that reason I have created one job and two transformations, given that in pentaho every step is executed in parallel.
The first transformation is going to be on charge of setting the variable and the second transformation is going to use this result as an input.
But in the first transformation I am having troubles by setting the variable, I do not understand where do I have to instanciate this variable to implement the "set season variable" step. And then how to get this result in the next transformation.
If anyone knows about this, or if you could recommend any link with a good example, I'll really appreciate it.
This can indeed be confusing for SSIS users. In PDI, you don't create a recordset variable as you do in SSIS. Simply creating a job creates one for you. Each job has two different types of "Results". One for recordset rows and one for filenames.
These variables are not directly accessible; they are just part of the job. There are steps that interact with them directly. For example under the "Job" branch when you're creating a transform, there is a Get rows from results step and a Copy rows to results step. They work directly with the job's row results.
Be aware that you must manually manage the metadata for the results. This is a pain, but over-all I find PDI's method of doing this more intuitive and easier than SSIS. I find SSIS more flexible in this regard.
There are also Get files from result and Set files in result. These interact with the job's built in file results. This is simply a list of every file touched by any step configured in the job. On the job tab there are tasks that deal with it directly such as Process result filenames, Add filenames to result and Delete filenames from results. These tasks operate on the built in file results list for the job and provide an easy way to, say, archive all the files loaded by the transform you just ran.
Be aware when using these steps that they record EVERY file touched by EVERY step in the job. If you look through most of the steps in transformations (data flows) that deal with files, there's usually an "Add files to results" checkbox that is checked by default. If you uncheck this, it will not add the file names to the jobs file results. You can also delete specific files from the file results with the Delete filenames from result step.
From your Job, start a Transformation:
Overload transformation variable into global variable in your job and use it:
I have a SSIS Package that I have a For Each Loop which imports multiple txt files into a SQL Server table. That runs fine.
What I am trying to accomplish is to store the distinct filename and date it was imported into a separate table. I created a separate For Each Loop for this and then archive the txt file after it's complete with a File System Task.
The issue I am having is I put an event handler to invoke a SQL Task and Send Email task if there is a warning (I was hoping for a warning only if there were no files in the directory where the package is importing from).
However, I found a warning that a column in the Data Flow task was not being used and should be removed if not needed. But the Data Flow task requires at least one field for me to put a Derived Column task
Derived Column Field1: pulls the #User: CurrentFile from the ForEachLoop Container.
Field2 pulls the current date.
Is there a way to perform this without the warning?
It sounds like you're over-complicating thing.
You have a ForEach loop and you're therefore assigning a value into some Variable to contain the file name, #User::CurrentFile. You can get the date it was loaded through either a call to GETDATE() or reference the system scoped variable, StartTime #[System::StarTime]
The most straight forward option would be to add an Execute SQL Task wired up to the OnSuccess Precedent Constraint from your Data Flow Task. The Execute SQL Task will then have a statement like INSERT INTO dbo.MyLog(FileName, InsertDate) SELECT ?, ?, assuming OLE DB Connection Manger, and then you map in your two variables.
Easy, clean, no warnings fired about unused columns in your data flow.
What I think you have is something like this, based on
I created a separate For Each Loop for this
Can somebody please help me to transfer around 15 tables from one database to another database. At present I can do this one by one using Data Flow task, but then I need to do this task 15 times which is very time consuming.
Why don't you just use a task? Maybe tasks->export is what you're looking for.
Otherwise you'll need to create separate blocks for each table or:
Create a variable of type object
Script Task: Add to your list all table names.
Iterate over this object variable with For each loop container
Inside the loop create a source from a variable. In this variable specify the connection dynamically depending on the current loop value.
you can use SSIS package, select Transfer SQL server objects from SSIS toolbox , in Object specify the source and destination servers and database. for copyAllObjects make it false . ObjectToCopy select CopyAllTables true or make it false and pick from the list the table you want to copy.
I have some requirements as explained below:
I need to have a SSIS data data flow task,which will accept the input parameters as
--SourceServer Connection String
--DestinationServer Connection String
--Source Table Name
--Destination Table Name
Then It should do the column mapping of source and destination tables dynamically at run time.
The source and destination tables schema would be same always.
I want to call this package by passing the above parameters from C#.net.
One main thing is here I will have to pass different sets of source and destination tables.
just answered this on a previous question. You cant loop through tables and dinamically map columns on your source and destinations components. You would need one set of Source -> Ddestination per table.
If that's not feasible, you may want to lokk at the Transfer SQL Server Objects Task
Create SSIS Packages parameters. Set web.config file for passing that parameters.
First you deploy the package in SQL Server.
Create one job for execute the package.
Create one sp using SQL Server.
& execute the job.
using sp_start_job.
I think it solve ur problem.