I want to synchronize two tables src,dest(DB SOURCE>> src table, DB Destination>> dest table) using ssis, where any (insert, update and delete operations on src will be applied to dest)
How can I achieve this using lookup tranformation ?
Thanks in advance.
Take table Dest in the lookup cache and then you need to lookup with Table Src .Choose option of the lookup as Redirect the non matching records. For the non matching records(not present in Table Dest) which are present in Table Src use OLE DB Destination to insert them in Table Dest.
For matching record use a physical table or temp table, use Execute SQl Task after the DFT to update those records in Table Src.
To speed up process try to use Cache Transform
Also you can achieve same by using Merge by following this article, Synchronize two tables using SSIS
Related
I want to be able to update a specific column of a table using data from another table. Here's what the two tables look like, the DB type and SSIS components used to get the tables data (btw, both ID and Code are unique).
Table1(ID, Code, Description) [T-SQL DB accessed using ADO NET Source component]
Table2(..., Code, Description,...) [MySQL DB accessed using ODBC Source component]
I want to update the column Table1.Description using the Table2.Description by matching them with the right Code first (because Table1.Code is the same as Table2.Code).
What i tried:
Doing a Merge Join transformation using the Code column but I couldn't figure out how to reinsert the table because since Table1 has relationships i can't simply drop the table and replace it with the new one
Using a Lookup transformation but since both tables are not the same type it didn't allow me to create the lookup table's connection manager (which would be for in my case MySQL)
I'm still new to SSIS but any ideas or help would be greatly appreciated
My solution is based on #Akina's comments. Although using a linked server would've definitely fit, my requirement is to make an SSIS package to take care of migrating some old data.
The first and last are SQL tasks, while the Migrate ICDDx is the DFT that transfers the data to a staging table created during the first SQL task.
Here's the SQL commands that gets executed during Create Staging Table :
DROP TABLE IF EXISTS [tempdb].[##stagedICDDx];
CREATE TABLE ##stagedICDDx (
ID INT NOT NULL,
Code VARCHAR(15) NOT NULL,
Description NVARCHAR(500) NOT NULL,
........
);
and here's the sql command (based on #Akina's comment) for transferring from staged to final (inside Transfer Staged):
UPDATE [MyDB].[dbo].[ICDDx]
SET [ICDDx].[Description] = [##stagedICDDx].[Description]
FROM [dbo].[##stagedICDDx]
WHERE [ICDDx].[Code]=[##stagedICDDx].[Code]
GO
Here's the DFT used (both TSQL and MySQL sources return sorted output using ORDER BY Code, so i didnt have to insert Sort components before the Merge Join) :
Note: Btw, you have to setup the connection manager to retain/reuse the same connection so that the temporary table doesn't get deleted before we transfer data to it. If all goes well, then after the Transfer Staged SQL Task, the connection would be closed and the global temporary table would be deleted.
how can we insert new data or update the data from one table to another table from MySQL to SQL server using ssis and by not using lookup.
A common way to do this is to insert new data to an empty temporary table, and then run SQL Merge command (using separate SQL Query task).
MERGE command is super powerful and can do updates, inserts or even deletes. See full description of Merge here:
https://learn.microsoft.com/en-us/sql/t-sql/statements/merge-transact-sql?view=sql-server-2017
The design for this will look like below :
You will have 4 tables and 1 view : Source, TMP_Dest (exactly as source with no PK), CHG_Dest(for changes, exactly as destination with no PK), Dest(will have PK), FV_TMP_Dest (this is in case the destination looks different than the source - different field types)
SSIS package :
1.Use ExecuteSQLTask and truncate TMP_Dest because it is just temporary for the extracted data
Use ExecuteSQlTask and truncate CHG_Dest because it is just temporary for the extracted data
Use one DataFlowTask for loading data from Source to TMP_Dest
Define two variables OperationIDInsert=1 and OperationIDUpdate=2 (the values are not important, you can set them as you want) -> you will use them at 5. point below
Use another DataFlowTask in which you will have:
on the left side OLE DB Source in which you will extract data from the view, ordered by PK (do not forget to set the SortKeyPosition from Advanced Editor for the PK fields)
on the right side OLE DB Source in which you will extract data from the Dest ordered by PK (do not forget to set the SortKeyPosition from Advanced Editor for the PK fields)
LEFT JOIN between this
on the left side ( "insert side") you will have: a derived column in which you will assign as Expression the OperationIDInsert variable AND an OLE DB Destination for inserting the data in CHG_Dest table. In this way, you will insert the data that have to be inserted in the destination table and you know this because you have the OperationIDInsert column.
on the right side you will do the same thing but using OperationIDUpdate column
You will use ExecuteSQLTask in the ControlFlow and will have an SQL Merge. Based on the PK fields and OperationIDInsert/OperationIDUpdate fields you will either insert the data or update it.
Hope this will help you. Let me know if you need additional info.
I have database table of 100,000 rows, imported from CSV each week using an SSIS package.
Usually updates, but sometimes it can add rows.
I see a few exceptions with staging table during the update rows - I don't know why? and how to update from staging to destination table?
This is the merge code :
MERGE INTO [PWCGFA_BG].[dbo].[Bank_Guarantees] WITH (HOLDLOCK) AS bg
USING [PWCGFA_BG].[dbo].[stagingBG] AS stgbg
ON bg.IATA_CODE = stgbg.IATA_CODE
WHEN MATCHED THEN
UPDATE set
bg.LEGAL_NAME=stgbg.LEGAL_NAME,
bg.TRADING_NAME=stgbg.TRADING_NAME,
bg.COUNTRY=stgbg.COUNTRY,
bg.CURRENCY=stgbg.CURRENCY,
bg.LANGUAGE=stgbg.LANGUAGE,
bg.STATUS=stgbg.STATUS,
bg.BANK_NAME=stgbg.BANK_NAME,
bg.BANK_GUARANTEE_AMOUNT=stgbg.BANK_GUARANTEE_AMOUNT,
bg.BANK_GUARANTEE_CURRENCY=stgbg.BANK_GUARANTEE_CURRENCY,
bg.BANK_GUARANTEE_EXPIRY_DATE=stgbg.BANK_GUARANTEE_EXPIRY_DATE,
bg.ACCREDITATION_DATE=stgbg.ACCREDITATION_DATE,
bg.CLASS_PAX_OR_CGO=stgbg.CLASS_PAX_OR_CGO,
bg.LOCATION_TYPE=stgbg.LOCATION_TYPE,
bg.XREF=stgbg.XREF,
bg.IRRS=stgbg.IRRS,
bg.TAX_CODE=stgbg.TAX_CODE,
bg.COUNTRY_CODE=stgbg.COUNTRY_CODE,
bg.CITY=stgbg.CITY,
bg.DEF=stgbg.DEF,
bg.OWN_SHARE_CHANGE=stgbg.OWN_SHARE_CHANGE
WHEN NOT MATCHED BY bg THEN
INSERT (IATA_CODE,LEGAL_NAME,TRADING_NAME,COUNTRY,CURRENCY,LANGUAGE,STATUS,BANK_NAME,BANK_GUARANTEE_AMOUNT,BANK_GUARANTEE_CURRENCY,BANK_GUARANTEE_EXPIRY_DATE,ACCREDITATION_DATE,CLASS_PAX_OR_CGO,LOCATION_TYPE,XREF,IRRS,TAX_CODE,CITY,DEF,OWN_SHARE_CHANGE)
VALUES (stgbg.IATA_CODE,stgbg.LEGAL_NAME,stgbg.TRADING_NAME,stgbg.COUNTRY,stgbg.CURRENCY,stgbg.LANGUAGE,stgbg.STATUS,stgbg.BANK_NAME,stgbg.BANK_GUARANTEE_AMOUNT,stgbg.BANK_GUARANTEE_CURRENCY,stgbg.BANK_GUARANTEE_EXPIRY_DATE,stgbg.ACCREDITATION_DATE,stgbg.CLASS_PAX_OR_CGO,stgbg.LOCATION_TYPE,stgbg.XREF,stgbg.IRRS,stgbg.TAX_CODE,stgbg.CITY,stgbg.DEF,stgbg.OWN_SHARE_CHANGE)
WHEN NOT MATCHED BY stgbg THEN
DELETE
If your source(staging) and destination tables on the same Server you can use MERGE statement with Execute SQL task, which is faster and very effective than a lookup which uses a row by row operation.
But if the destination is on a different Server, you have the following options
Use lookup to update the matching rows with an OLEDB Command(UPDATE Statement)
Use a Merge Join (with LEFT OUTER JOIN) to identify the new/matching records and then use a conditional split to INSERT or UPDATE records. This works same as the Lookup but faster.
Create a temporary table in the destination db, dump data from staging to that table and then use the MERGE statement, this is faster than using a lookup.
Same database exist on two servers. I will call them Source and Destination for simplicity here. I need to compare records and delete those only exist in target but not in Source. I can't use Execute SQL as both databases exist on different Servers and there is no link between them. Can anyone propose a solution?
You could populate a table on the target with the IDs from the source table. Then use an Execute SQL task on the Destination database to delete rows from target table that do not exist in the table you populated. Eg, something like:
DELETE FROM TargetTable
WHERE ID NOT IN (SELECT ID FROM TableIDsFromDestination)
What you can do is create an Staging base in the Destination server where you are going to copy the information without look for references, after that you can do the compare directly in the same server.
I hope it helps.
I have to pull data from a SQL database table to my DB2 table. If records already exist UPDATE, for new records INSERT, for extra records in destination table DELETE those extra records. Destination table looks exactly like source table. For INSERT/UPDATE I am fine, how do I do DELETE from dest table?
DB2 has a MERGE command. This allows you to write a single SQL statement to do an INSERT, UPDATE and DELETE based on conditions you define. It is a very clean way of doing this.
So what you will do is add an "Execute SQL Task" element to your SSIS package, and add the DB2 merge statement to the task.
See this link (at the bottom are examples) - http://publib.boulder.ibm.com/infocenter/db2luw/v9/index.jsp?topic=%2Fcom.ibm.db2.udb.admin.doc%2Fdoc%2Fr0010873.htm
if all you want is a copy of the source table... then avoid complexity and delete the target entirely first - then everything is just an insert.