I have an SSIS package, There are only two tasks in my SSIS package. One is Execute SQL Task and another one is Data Flow Task. My first task is doing to truncate a table (Table_1) and second task is just load data in truncated table means Table_1. An SSRS report get data from that table means Table_1. My SSIS packages run every hour. when SSIS package running in the same time users has complaining that they are not able to view data in the report. How do I do that my SSIS package start running. Users can able to view data in my Report.
Off the top of my head, have a table that stores values on whether this deal is running or not, doing something like setting a single value to 1 when running and 0 when not running.
Then in your SSRS, instead of a 'SELECT * FROM Table_1', write a stored procedure that has logic to lookup the value in the above table. If 0 return the set normally, if 1 then return some kind of friendly message that data is currently being loaded and the report is not available.
Related
Imagine that you want to save in a variable the number of rows the were updated or deleted in a table.
This is the steps that i did:
First, in the Control flow i created a Data Flow Task.
Them, in the Data Flow, i created a source(in my case is a excel file), then i proceeded to create two variables to count those rows- countDeleted and countUpdated, then connected the variables to two row count transformations, and them connected my destination (OLE DB).
Now in the control flow, what do i do??
Create a SQL execute task?? or a Script task?? What is the best way to do it?? What is the piece of code to use??
Thanks for youy help.
PS: i only have 4 weeks off SSIS, sorry for my noobieness :)
An OLD DB destination only inserts. It can't UPDATE or DELETE
What's your logic for updating or deleting?
If you're just starting out and reading about doing things in SSIS you will eventually find advice to use the OLE DB Command to perform row by row delete and inserts.
In my opinion this is to be avoided. It does not scale (works fine for small recorsets then fails for large recordsets), and it is difficult to maintain parameter mappings in the OLE DB Command. Although you should try it anyway to familiarise yourself with it.
My advice is to load the Excel data into a staging table, perform batch DELETE and UPDATE statements to load the data and use ##ROWCOUNT to capture the records updated.
For example;
Your existing described dataflow can be used to load into a table called StagingTable
Before your dataflow you should run an Execute SQL Task (This is in the Control Flow pane, not the Data Flow pane) that clears the staging table:
TRUNCATE TABLE StagingTable;
So first get that working - repeatedly running your package clears the staging table then loads Excel into it without creating duplicates
This in itself is a challenge as Excel is a terrible data interchange format.
Once you have that working, you add an execute SQL task to the end that runs some SQL that deletes the records you want and captures the count. For example:
DELETE FROM MyFinalTable WHERE PriamryKey IN (SELECT PrimaryKey FROM StagingTable);
SELECT ##ROWCOUNT;
Then you follow the instructions here to load that back to your SSIS variable
http://microsoft-ssis.blogspot.com/2011/03/rowcount-for-execute-sql-statement.html
What are you doing with this row count? Are you writing it to a logging table? Save
yourself the bother of pulling it back into an SSIS variable and just write it directly:
DELETE FROM MyFinalTable WHERE PriamryKey IN (SELECT PrimaryKey FROM StagingTable);
INSERT INTO LogTable(Table,Operation,Type)
SELECT 'MyFinalTable','Delete', ##ROWCOUNT;
In my experience it is not a good idea to build convoluted logic into SSIS packages if you can instead do in a database. Although it does depend on the person who has to eventually maintain it. Hopefully you can appreciate that this T-SQL approach is a more straightforward code based approach as opposed to having to dig around in property pages and events and other places inside SSIS packages.
I assume that you're using an Execute SQL Task for the updates and deletes? As #Nick.McDermaid mentioned, using an OLE DB Command within a Data Flow presents various issues when performing DML. You can find the number of rows updated, inserted, or deleted in a table through an Execute SQL Task by using the ExecValueVariable property of this task. Set the variable that will hold the row count to this property and it will return the number of affected rows. Note that is will only return the number of rows impacted by the last statement in the Execute SQL Task, regardless of batches (i.e. GO separators) are in the component.
I have a number of SSIS packages that run net change import processes from one database to another.
At the destination server data is inserted into a temp table by a data flow task, and then I run a very simple set of net change insert/update/delete queries in an Execute SQL task.
I'd like to report the number of rows inserted, deleted and updated back to SQL agent - specifically I'd like these counts to show up in the "view history" interface in SSMS for any jobs running this SSIS package.
I feel like this should be incredibly simple (and it's no doubt going to turn out to be), but I can't work out how to do it.
I'm running SQL Server 2008 R2
It's working with CmdExec and T-SQL job type. Give it a try for SSIS job type.
In Execute SQL task, store count of updated/inserted/deleted rows. Use PRINT statements in the task. Then go to New Job => Steps => New => under New Job Step click Advanced pane, and check the "Include step output in history" checkbox.
I am using SSIS 2012 with a data flow task having a data source and an Ole DB Sql Task.
The data source is creating a set of Id's { 1,2,3, etc } with the Ole DB Sql Task deleting a record in another database-table. What I am seeing in the Sql Profiler is a delete command for each Id which is expected as it is working on a record by record basis. I can get upto 10,000 records.
Is there any way I work with the output of the data source as a set and say:
delete from Table1 where Id in { set of Id's }
You cannot do that in SSIS.
In fact, you can build an expression and execute that expression in SSIS, but you don't WANT to do that. Expressions are limited in the number of characters they can have, and they are a mess at maintenance time.
Some things are better done directly in a stored procedure, while other are better in SSIS. The art of SSIS is to know when to do it in SSIS or in a procedure.
Good luck!
I am working with a SQL Server 2008 R2 BIDS 2008 solution.
I need to transfer data from an Oracle(11g) source database to a SQL Server database. I need to copy all data from multiple tables in the Oracle database. I have a ForEach container in which there is one Data Flow Task.
I want to loop round a list of table names, select all columns out of in the Oracle source and copy to dest_ in the SQL Server.
I have added a Property Expression to the Data Flow Task for [ADO NET Source].[SqlCommand] with an expression of "SELECT * FROM " + #[User::ImportPath]
As the table is undefined at this point it isn't clear if I can map columns or setup the ADO NET Destination task correctly.
Is there steps I am missing? Is what I am attempting to do even possible?
You cant loop through tables and dinamically map columns on your source and destinations components. You would need one set of Source -> Ddestination per table.
If that's not feasible, you may want to lokk at the Transfer SQL Server Objects Task
I have a very simple Data flow task reading data from a FF and inserting the data in a table. At the same time I would like to write in an Audit table, how many rows have been inserted, the created date...
How can I do that easily?
If you are interested only in the number of rows being successfully processed or number of rows that encountered errors, then you can make use of in-built SSIS logging feature. Please check the below mentioned steps. I hope the example gives you an idea. I have displayed only two columns from the log table but there are other useful fields like starttime, endtime etc., The example was created in SSIS 2008 R2
Click on the SSIS package.
On the menus, select SSIS --> Logging...
On the Configure SSIS Logs: dialog, select the provider type and click Add. I have chosen SQL Server for this example. Check the Name checkbox and provide the data source under Configuration column. Here SQLServer is the name of the connection manager. SSIS 2008 or SSIS 2008 R2 will create a table named dbo.sysssislog and stored procedure dbo.sp_ssis_addlogentry in the database that you selected. Refer screenshot #1 below. The table name in SSIS 2005 is dbo.sysdtslog90 and the stored procedure is named as dbo.sp_dts_addlogentry
If you need the rows processed, select the checkbox OnInformation. Here in the example, the package executed successfully so the log records were found under OnInformation. You may need to fine tune this event selection according to your requirements. Refer screenshot #2 below.
Here is a sample package execution within data flow task. Refer screenshot #3 below.
Here is a sample output of the log table dbo.sysssislog. I have only displayed the columns id and message. There are many other columns in the table. In the query, I am filtering the output only for the package named 'Package1' and the event 'OnInformation'. You can notice that records with ids 7, 14 and 15 contain the rows processed. Refer screenshot #4 below.
Hope that helps.
Screenshot #1:
Screenshot #2:
Screenshot #3:
Screenshot #4:
You can multicast the flat file, or use a trigger on the table you're inserted data to. If it's a table that's being audited you'll probably want to know whenever any data is inserted.