I'm new to SSIS and am writing a package that includes moving data to a table that is created in a previous Execute SQL Task object.
The issue that I'm encountering is that I am unable to create a data flow destination task that uses a dynamic destination table name.
The intended process is:
Execute SQL Task object creates new table based on today's date (i.e. Table1_20111014)
Data Flow task moves data from table "Table1" to "Table1_20111014".
The column metadata for Table1 and Table1_20111014 are the same, and does not change. However, the name of the table the data needs to be moved to will change depending on the date at time of execution.
Is it possible to dynamically specify the destination table in a destination data flow object?
If not, are there known workarounds or is using SSIS for this task a bad idea?
As long as the meta data remains the same, there is no drawback to using dynamic destination table name.
To accomplish this, on the ole db destination instead of using "table name" or "table name fast load" use the equivalent "from variable" table load option. This obviously assumes you have a variable defined that contains the name of the table created in the execute sql task
Related
I have used a Lookup in my data flow task. When I use Full Cache mode, the data flow task runs fine. But when I use Partial Cache or no Cache in my lookup, the records do not go past the lookup task and it keeps running for hours. I have checked for errors but there aren't any errors displayed. Could anyone please help me on this?
A Lookup is not appropriate for your task. Instead:
Add an OLE DB Source to pull in the data
Sort the records from the incoming source and the OLE DB Source
Perform a merge join (Full outer).
Add a Derived Column Transformation to check for ISNULL on the two joining columns. Create a new output column Called Action. For the NULLs in the target then you will tag that as an INSERT record.
Add a conditional split to send the INSERT record to an OLE DB Destination to insert the new records.
You can also check to see if there are matches between the two populations and perform updates, or look for NULLs in the source and DELETE in the destination.
I have various data inflows from different servers. I need to populate all the data in one single destination table.
I need a staging table inside the data flow task where i can insert all the data and do some validations and then insert those data directly to destination table.
Is this possible ? Also i need to pull only the recent records based on the modified date column of the source table. (I'm trying to create a variable that stores the date of the recent successful package execution which I can compare it with Modified date column) Is this possible ?
Let me know is there any best way to achieve above all process.
I don't know that it's possible to do this within one data flow task, but couldn't you use one data flow task for each stage and link them via control flow instead?
It sounds to me like the best way to do what you want is to import the data and then run a SQL task on the table with the imported data (control flow object in SSIS), it should be easy to filter the source table as you mentioned:
SELECT *
INTO newTable
FROM sourceTable
WHERE DateModified >= somedate
Hope that is useful.
I have a SSIS Package that I have a For Each Loop which imports multiple txt files into a SQL Server table. That runs fine.
What I am trying to accomplish is to store the distinct filename and date it was imported into a separate table. I created a separate For Each Loop for this and then archive the txt file after it's complete with a File System Task.
The issue I am having is I put an event handler to invoke a SQL Task and Send Email task if there is a warning (I was hoping for a warning only if there were no files in the directory where the package is importing from).
However, I found a warning that a column in the Data Flow task was not being used and should be removed if not needed. But the Data Flow task requires at least one field for me to put a Derived Column task
Derived Column Field1: pulls the #User: CurrentFile from the ForEachLoop Container.
Field2 pulls the current date.
Is there a way to perform this without the warning?
It sounds like you're over-complicating thing.
You have a ForEach loop and you're therefore assigning a value into some Variable to contain the file name, #User::CurrentFile. You can get the date it was loaded through either a call to GETDATE() or reference the system scoped variable, StartTime #[System::StarTime]
The most straight forward option would be to add an Execute SQL Task wired up to the OnSuccess Precedent Constraint from your Data Flow Task. The Execute SQL Task will then have a statement like INSERT INTO dbo.MyLog(FileName, InsertDate) SELECT ?, ?, assuming OLE DB Connection Manger, and then you map in your two variables.
Easy, clean, no warnings fired about unused columns in your data flow.
What I think you have is something like this, based on
I created a separate For Each Loop for this
I am importing an Excel table from an ftp site to SQL using SSIS. The destination table is going to be used to calculate good and bad records based on another SQL database. Here is my problem. The Excel file is name RTW_032613_ABC_123.xls. This file name is a concatenation of a number of fields. I cannot recreate it based on the fields in the table, so I need to retain it and pass it to the new table in SQL. I have a parameter #FileName that I am using to loop through the files in the ftp folder. What I would like to do is either combine the import of data from the Excel file with the file name or insert the file name in each record after the import. I am calling the SSIS procedure from another stored procedure in SQL. I tried adding a SQL data flow task but I am not seeing where I add the insert statement on either the SQL Server Compact Destination or SQL Server Destination.
I am over my head with SSIS on this one. The key is that the parameter that I need is available in SSIS but I really need to get it passed on to my SQL table.
TIA
If I'm reading your question right, you have an SSIS package with a variable containing the filename and you want to save the filename with each row that you are sending to your SQL table? If so:
Add a derived column to the data flow, making a new column and referencing the variable in the expression
Include that new column in the mapping for the destination of your data flow, sending the filename to whichever column you would like to save that data in.
No need for a seperate SQL task.
I have two OLEDB Data Sources that have similar columns:
TMP_CRUZTRANS
-------------
CUENTA_CTE numeric (20,0)
TMP_CTACTE_S_USD
----------------
CON_OPE numeric(20,0)
I need to substract all the similar values between this two tables and keep the rows which are different. Is there a transformation/task within SSIS that can perform NOT IN constraint normally used in SQL query?
Currently, I am performing this operation using Execute SQL Task on Control Flow.
The top Data Flow creates the first table TMP_CRUZTRANS (Merge join between other 2 tables... But I guess that's not important for my question) that i need to keep the different values with the second table.
In the Execute SQL Task, I have the following statement:
INSERT INTO [dbo].[TMP_CYA]
SELECT RUT_CLIE, CUENTA_CTE, MONTO_TRANSAC
FROM [dbo].[TMP_CRUZTRANS]
WHERE CUENTA_CTE NOT IN (SELECT CON_OPE FROM TMP_CTACTE_S_USD)
Finally, with the new table TMP_CYA I can continue with my work.
The problem with this approach is that the TMP_CRUZTRANS got like 5 millions of rows, so it's VERY slow inserting all this data into a table using Execute SQL Task. It takes about like 5 hours to perform this operation. That's why I need to do this inside the Data Flow task.
You can use Lookup transformation available within Data Flow task to achieve your requirement.
Here is a sample that illustrates what you are trying to achieve.
Create a package with data flow task. Inside the data flow task, use OLE DB Source to read data from your source table TMP_CRUZTRANS. Use Lookup transformation to validate the existence of the values against the table dbo.TMP_CTACTE_S_USD between given columns. Then redirect the non-matching output to OLE DB Destination to insert rows into table dbo.TMP_CYA
Here is how data flow task would look like in place of the Execute SQL Task that you are currently using.
Configure the Lookup transformation as shown below:
On the General tab page, select Redirect rows to no match output from Specify how to handle rows with no matching entries because you are interested only in non matching rows.
On the Connection tab page, select the appropriate OLE DB Connection manager and select the table dbo.TMP_CTACTE_S_USD. That is the table against which you would like to validate the data.
On the Columns tab page, drag the column CUENTA_CTE and drop it on CON_OPE to establish the mapping between source and lookup tables. Click OK.
When you connect the Lookup transformation with OLE DB Destination, Input Output Selection dialog will appear. Please make sure to select Lookup No Match Output.
Here is the sample before executing the package.
You can see that only 2 rows non matching rows have been transferred to OLE DB destination.
You can notice that the destination table now contains the two non matching rows after package execution.
Hope that helps.