Reading incremental data based on the value of composite Primary key - mysql

Question

I have an OLTP (source) from where data has to be moved to the DWH(destination) on an incremental basis.
The source table has a composite Primary Key on Loan_id, AssetID as shown below.
LOAN_ID, ASSETID, REC_STATUS
'12848','13170', 'F'
Had it been a single col primary key then I would check for the max value of the column at the destination and then read all the records from the source where the Primary key value is greater than the max value at the destination, but as it is a composite primary key, this will not work.
Any idea how this can be done using T-SQL Query?
Specs: Source is an MYSQL DB and the destination is MSSQL 2012. The connection is made using a linked server.

Answer 1

There's a few things that you can try. Dealing with a linked server and not knowing the specifics of that setup, along with volume of data, performance could be an issue.
If you're not worried about changes in existing records or deletes, a simple left outer join will get you any records that haven't been inserted into your destination yet:
SELECT [s].[LOAD_ID]
, [s].[ASSETID]
, [s].[REC_STATUS]
FROM [LinkedServer].[Database].[schema].[SourceTable] [s]
LEFT OUTER JOIN [DestinationTable] [d]
ON [s].[LOAN_ID] = [d].[LOAN_ID]
AND [s].[ASSETID] = [d].[ASSETID]
WHERE [d].[LOAN_ID] IS NULL;
If you're worried about changes you could still use a left outer and look for NULL in destination or differences in field values, but then you'd need an additional update statement.
SELECT [s].[LOAD_ID]
, [s].[ASSETID]
, [s].[REC_STATUS]
FROM [LinkedServer].[Database].[schema].[SourceTable] [s]
LEFT OUTER JOIN [DestinationTable] [d]
ON [s].[LOAN_ID] = [d].[LOAN_ID]
AND [s].[ASSETID] = [d].[ASSETID]
WHERE [d].[LOAN_ID] IS NULL --Records from source not in destination
OR (
--This evaluates those in the destination, but then checks for changes in field values.
[d].[LOAN_ID] IS NOT NULL
AND (
[s].[REC_STATUS] <> [d].[REC_STATUS]
OR [s].[SomOtherField] <> [d].[SomeOtherField]
)
);
--The above insert into some landing or staging table on the destination side and then you could do a MERGE.
If we need to worry about deletes. A record was deleted from the source and you don't want it in the destination anymore. Flip the left outer to find records in your destination no longer in the source:
DELETE [d]
FROM [DestinationTable] [d]
LEFT OUTER JOIN [LinkedServer].[Database].[schema].[SourceTable] [s]
ON [s].[LOAN_ID] = [d].[LOAN_ID]
AND [s].[ASSETID] = [d].[ASSETID]
WHERE [s].[LOAD_ID] IS NULL;
You could attempt doing all of this using a merge. Try the MERGE over a linked server or bring all the source records to the destination in a land/stage table and then do the merge there. Here's an example of attempting over a linked server.
MERGE [DestinationTable] [t]
USING [LinkedServer].[Database].[schema].[SourceTable] [s]
ON [s].[LOAN_ID] = [d].[LOAN_ID]
AND [s].[ASSETID] = [d].[ASSETID]
WHEN MATCHED THEN UPDATE SET [REC_STATUS] = [s].[REC_STATUS]
WHEN NOT MATCHED BY TARGET THEN INSERT (
[REC_STATUS]
)
VALUES ( [s].[REC_STATUS] )
WHEN NOT MATCHED BY SOURCE THEN DELETE;
When dealing with a merge, you gotcha watch out for this statement:
WHEN NOT MATCHED BY SOURCE THEN DELETE;
If you're not working with the entire record set you could lose records in you destination. Example would be, you've limited the results set you pulled from the source into a staging table, you now merge the staging table with the final destination, anything outside of that would get deleted in your destination. You can solve that by limiting your target with a CTE, Google: "merge into cte as the target". That's if you have a date you can filter on.
If you have a date column, that's always helpful, especially some sort of Change/Update date column when new records are inserted or updated. Then you can filter on your source to only those records you care about.

Answer 2

Incremental loads typically have a date driving them.
You can use a composite key inside a lookup. This has been answered many times.
Add a lookup and change the test to redirect no match (default is fail).
Basically you check if the key exists in the destination.
If the key exist then it is an update (match).
If the key does not exist (no match) then it is an insert.