I'm using a data flow task and 2 Ole Db sources. The 2 sources bring in data from tables on 2 different databases on the same server. The 2 tables can be mapped by ids. All of the ids from the second table (closedstops) exist in the first table (stops). I need to remove all the the closed stops by id from the first table. Afterwards I need to export the first table out of the database into a text file.
Do I need to use a merge join before deleting or do I need to use a ole db command to delete records (see attached screenshot). I have looked at many questions and answers on stackoverflow as well as tutorials and none of them quite answer my question. Any help is greatly appreciated. Thank you.
Closed stops is the driver table. Leave it be.
Instead of an OLE DB Source table for "stops" change that to a Lookup Component. You are only interested in rows that match.
And then you can use your OLE DB Command to fire off single delete statements.
My preference for performance and traceability would be to insert all the "to be deleted" ids into a table on the Stops database. When the Data Flow has completed, an Execute SQL Task would then fire up to perform the deletes in a set based operation(s).
Related
Imagine that you want to save in a variable the number of rows the were updated or deleted in a table.
This is the steps that i did:
First, in the Control flow i created a Data Flow Task.
Them, in the Data Flow, i created a source(in my case is a excel file), then i proceeded to create two variables to count those rows- countDeleted and countUpdated, then connected the variables to two row count transformations, and them connected my destination (OLE DB).
Now in the control flow, what do i do??
Create a SQL execute task?? or a Script task?? What is the best way to do it?? What is the piece of code to use??
Thanks for youy help.
PS: i only have 4 weeks off SSIS, sorry for my noobieness :)
An OLD DB destination only inserts. It can't UPDATE or DELETE
What's your logic for updating or deleting?
If you're just starting out and reading about doing things in SSIS you will eventually find advice to use the OLE DB Command to perform row by row delete and inserts.
In my opinion this is to be avoided. It does not scale (works fine for small recorsets then fails for large recordsets), and it is difficult to maintain parameter mappings in the OLE DB Command. Although you should try it anyway to familiarise yourself with it.
My advice is to load the Excel data into a staging table, perform batch DELETE and UPDATE statements to load the data and use ##ROWCOUNT to capture the records updated.
For example;
Your existing described dataflow can be used to load into a table called StagingTable
Before your dataflow you should run an Execute SQL Task (This is in the Control Flow pane, not the Data Flow pane) that clears the staging table:
TRUNCATE TABLE StagingTable;
So first get that working - repeatedly running your package clears the staging table then loads Excel into it without creating duplicates
This in itself is a challenge as Excel is a terrible data interchange format.
Once you have that working, you add an execute SQL task to the end that runs some SQL that deletes the records you want and captures the count. For example:
DELETE FROM MyFinalTable WHERE PriamryKey IN (SELECT PrimaryKey FROM StagingTable);
SELECT ##ROWCOUNT;
Then you follow the instructions here to load that back to your SSIS variable
http://microsoft-ssis.blogspot.com/2011/03/rowcount-for-execute-sql-statement.html
What are you doing with this row count? Are you writing it to a logging table? Save
yourself the bother of pulling it back into an SSIS variable and just write it directly:
DELETE FROM MyFinalTable WHERE PriamryKey IN (SELECT PrimaryKey FROM StagingTable);
INSERT INTO LogTable(Table,Operation,Type)
SELECT 'MyFinalTable','Delete', ##ROWCOUNT;
In my experience it is not a good idea to build convoluted logic into SSIS packages if you can instead do in a database. Although it does depend on the person who has to eventually maintain it. Hopefully you can appreciate that this T-SQL approach is a more straightforward code based approach as opposed to having to dig around in property pages and events and other places inside SSIS packages.
I assume that you're using an Execute SQL Task for the updates and deletes? As #Nick.McDermaid mentioned, using an OLE DB Command within a Data Flow presents various issues when performing DML. You can find the number of rows updated, inserted, or deleted in a table through an Execute SQL Task by using the ExecValueVariable property of this task. Set the variable that will hold the row count to this property and it will return the number of affected rows. Note that is will only return the number of rows impacted by the last statement in the Execute SQL Task, regardless of batches (i.e. GO separators) are in the component.
I have one table. It contains 34mln rows. When I tried "INSERT...SELECT..." SQL query via SQL SERVER MANAGEMENT STUDIO. Now I am trying to copy it using SSIS. How to copy data of table into different other tables?
This is relational schema where I want to divide parts by columns and insert into these tables.
Just briefing, Cerate a SSIS Package. Which flows like follow
Take Data Flow Task
Take OLE DB Source and point to Your source table.
Keep one lookup to restrict already loaded recodrds.(If you execute multiple times)
Take a OLE DB Destination and point to parent table.
Take one more Data flow task and point to your source table.
keep lookup to get ParentTable ID
Keep one lookup to restrict already loaded recodrds.(If you execute multiple times)
Take OLE DB for 3 child tables and Load into your child tables.
I was creating 10 tables with same schema for loading application LOG.
I was creating 10 tables with same schema and at the end I do create a alias name which will represent all those 10 tables.
So it take care about current active table selection to insert the records, so once table1 gets filled more records it will choose table2 and next table3....
So while selecting a record, it will check all the 10 tables for the requested record.....
Now my question is, I forgot about how did I created and which statement I have used.
If anyone knows then please let me know about this.
You are reinventing partitioning, please use the tools MySQL provides for this, it will save you a lot of time & headaches.
I am facing a situation where I have built SSIS package for consolidating a table from multiple databases but sometimes DBA manually deletes records from source tables. Is there any way to detect on consolidated table if a record was deleted? I know that I can truncate consolidated table each time before the package runs but I wonder if there's another way?
I demonstrate one way to manage deleting data in the destination that has been deleted in the source in the Stairway to Integration Services series at SQL Server Central.
Hope this helps,
Andy
I thought of several solutions, but I realize now the big issue is that you don’t have an unified data source, so a record may not exist in A but exist in B, thus you can’t delete it.
So my suggestion is, after you inserted, run a select distinct(id) from all sources and put them together on a structure (table, temp table, view, UNION, whatever..) and run a Merge from this source into destination on the IDs with only a WHEN NOT MATCHED THEN DELETE.
I am new to SSIS, so please bear with me.
I created an Integration Services Project for SQL Server 2008 to import data from an old db to a new one. One of the things I need to do is import data from two old source tables into one new destination table.
What is the best way to do this?
I can easily see the results I want with a simple inner join query using tsql, but am not having any luck using the SSIS package. My current approach is a three step process:
Add OLE DB Source component that pulls all columns from my first source table
Add a Lookup component, which is the next step after my OLE DB Source component. In this I query the second source table 'using the results of a sql query' that returns no nulls, then drag the foreign key id from the 'available input columns' to the primary key in the available lookup columns. I also check the checkboxes in 'available input columns' to add 2 more columns.
Add OLE DB Destination, pointed to my destination table.
This process fails at the first step, not at the lookup step, and fails with the error "Row yielded no match during lookup". The foreign key cannot be null, and obviously the primary key can't either. I used a SQL statement in step to so I could make sure I don't get any null date values in the columns (there were a few) but I am still getting the error. If I output the first step failure path to a Flat File Destination, I get an empty CSV (watching in debug mode says ~600k records go into the flat file).
I am pretty stumped at this point and this seems like it should be super easy task. I have scoured the web for answers, and found this link that sounds like the same exact problem I am having, but changing the cache setting didn't help.
Help appreciated!
It sounds like you have a mismatch in the lookup. I'd hand run the queries and verify that tha both OLE DB SOurce has no null foriegn keys; and that each foreign-key matches something in the lookup table.
There is a simpler approach here. Use your inner join query you mentioned in the OLE DB SOurce. Don't use the table select, provide your SQL query with the join. This let's the SQL Server do all of the heavy lifting of the join and then SSIS can do the transferring.