I have a simple data flow in SSIS (defined in visual studio 2013), which uses SQL to extract data from one sql server instance's table A to then add it to another SQL server instance's table B.
What is the best practice pattern to truncate the data in table B? A truncate statement like this:
TRUNCATE TABLE B
after the select statement for table A - especially when you have have a fairly big table to 'transmit'?
One thing I have done in cases like that is to create two copies of the same table and then a view that points to one or the other that has the name of the current table.
The SSIS package then determines which table is in use and sets the connection for the table to populate to the other table.
Then an exec SQl task truncates the table not currently in use. You may also want to drop any indexes at this point.
Then a dataflow populates the table not currently in use.
Then recreate any indexes you dropped.
Finally an exec SQL task drops and creates the view to use the table you just populated instead of the other one.
Total down time of the table being referenced? Generally less than a second for the drop and create view no matter how long it takes to populate the table.
Related
I want to be able to update a specific column of a table using data from another table. Here's what the two tables look like, the DB type and SSIS components used to get the tables data (btw, both ID and Code are unique).
Table1(ID, Code, Description) [T-SQL DB accessed using ADO NET Source component]
Table2(..., Code, Description,...) [MySQL DB accessed using ODBC Source component]
I want to update the column Table1.Description using the Table2.Description by matching them with the right Code first (because Table1.Code is the same as Table2.Code).
What i tried:
Doing a Merge Join transformation using the Code column but I couldn't figure out how to reinsert the table because since Table1 has relationships i can't simply drop the table and replace it with the new one
Using a Lookup transformation but since both tables are not the same type it didn't allow me to create the lookup table's connection manager (which would be for in my case MySQL)
I'm still new to SSIS but any ideas or help would be greatly appreciated
My solution is based on #Akina's comments. Although using a linked server would've definitely fit, my requirement is to make an SSIS package to take care of migrating some old data.
The first and last are SQL tasks, while the Migrate ICDDx is the DFT that transfers the data to a staging table created during the first SQL task.
Here's the SQL commands that gets executed during Create Staging Table :
DROP TABLE IF EXISTS [tempdb].[##stagedICDDx];
CREATE TABLE ##stagedICDDx (
ID INT NOT NULL,
Code VARCHAR(15) NOT NULL,
Description NVARCHAR(500) NOT NULL,
........
);
and here's the sql command (based on #Akina's comment) for transferring from staged to final (inside Transfer Staged):
UPDATE [MyDB].[dbo].[ICDDx]
SET [ICDDx].[Description] = [##stagedICDDx].[Description]
FROM [dbo].[##stagedICDDx]
WHERE [ICDDx].[Code]=[##stagedICDDx].[Code]
GO
Here's the DFT used (both TSQL and MySQL sources return sorted output using ORDER BY Code, so i didnt have to insert Sort components before the Merge Join) :
Note: Btw, you have to setup the connection manager to retain/reuse the same connection so that the temporary table doesn't get deleted before we transfer data to it. If all goes well, then after the Transfer Staged SQL Task, the connection would be closed and the global temporary table would be deleted.
how can we insert new data or update the data from one table to another table from MySQL to SQL server using ssis and by not using lookup.
A common way to do this is to insert new data to an empty temporary table, and then run SQL Merge command (using separate SQL Query task).
MERGE command is super powerful and can do updates, inserts or even deletes. See full description of Merge here:
https://learn.microsoft.com/en-us/sql/t-sql/statements/merge-transact-sql?view=sql-server-2017
The design for this will look like below :
You will have 4 tables and 1 view : Source, TMP_Dest (exactly as source with no PK), CHG_Dest(for changes, exactly as destination with no PK), Dest(will have PK), FV_TMP_Dest (this is in case the destination looks different than the source - different field types)
SSIS package :
1.Use ExecuteSQLTask and truncate TMP_Dest because it is just temporary for the extracted data
Use ExecuteSQlTask and truncate CHG_Dest because it is just temporary for the extracted data
Use one DataFlowTask for loading data from Source to TMP_Dest
Define two variables OperationIDInsert=1 and OperationIDUpdate=2 (the values are not important, you can set them as you want) -> you will use them at 5. point below
Use another DataFlowTask in which you will have:
on the left side OLE DB Source in which you will extract data from the view, ordered by PK (do not forget to set the SortKeyPosition from Advanced Editor for the PK fields)
on the right side OLE DB Source in which you will extract data from the Dest ordered by PK (do not forget to set the SortKeyPosition from Advanced Editor for the PK fields)
LEFT JOIN between this
on the left side ( "insert side") you will have: a derived column in which you will assign as Expression the OperationIDInsert variable AND an OLE DB Destination for inserting the data in CHG_Dest table. In this way, you will insert the data that have to be inserted in the destination table and you know this because you have the OperationIDInsert column.
on the right side you will do the same thing but using OperationIDUpdate column
You will use ExecuteSQLTask in the ControlFlow and will have an SQL Merge. Based on the PK fields and OperationIDInsert/OperationIDUpdate fields you will either insert the data or update it.
Hope this will help you. Let me know if you need additional info.
I created a view using the Members table. If I truncate the Members table will it automatically truncate the view data?
A view is like a virtual table and does not have a Physical existence. A View is derived from an SQL Query which may pull data from a Single table or Multiple ones.
So when you do a query on a view, What it actually does is running the View Query on the background. So if you make any changes in the data on the Tables using the Underlying script, then the same will be effected automatically in the view
I have to create a particular system of reviews in which I should do a lot of AVG in my query, so I want to ask if it's possible to create a table in SQL in which I have these Averages automatically updated?
Mysql 5.5.24-0ubuntu0.12.04.1
Yes, there are two ways to do this:
You can create a "view", which is like a table, except that its records come from a SQL query (usually derived from other tables) rather than being stored directly. See http://dev.mysql.com/doc/refman/5.6/en/create-view.html.
You can create a "trigger", which is procedural code that runs whenever a specified table is modified in a specified way. In your case, you would create a trigger that runs whenever one table is modified, and that re-generates the secondary table. See http://dev.mysql.com/doc/refman/5.6/en/create-trigger.html.
What is the purpose of a temporary table like in the following statement? How is it different than a regular table?
CREATE TEMPORARY TABLE tmptable
SELECT A.* FROM batchinfo_2009 AS A, calibration_2009 AS B
WHERE A.reporttime LIKE '%2010%'
AND A.rowid = B.rowid;
Temp tables are kept only for the duration of your session with the sever. Once the connection's severed for any reason, the table's automatically dropped. They're also only visible to the current user, so multiple users can use the same temporary table name without conflict.
Temporary table ceases to exist when connection is closed. So, its purpose is for instance to hold temporary result set that has to be worked on, before it will be used.
Temporary tables are mostly used to store query results that need further processing, for instance if the result needs to be queried or refined again or is going to be used at different occasions by your application. Usually the data stored in a temporary database contains information from several regular tables (like in your example).
Temporary tables are deleted automatically when the current database session is terminated.
Support for temporary tables exists to allow procedural paradigms in a set-based 4GL, either because the coder has not switched their 3GL mindset to the new paradigm or to work around a performance or syntax issue (perceived or otherwise).