I'm really new to SSIS and I would like to automate some of my workflow. However, most of my tasks require that other jobs/packages in SQL Server Job Agent have previously been run for the day/week by other team members. My question is: how can I direct my workflow based on the status of these other packages? Control Flow doesn't have a Conditional Split tool, and I can't kick off a separate job/package from Data Flow. I can easily write a SQL statement to determine whether any or all of the related packages have been run, and even have it return a boolean value if needed, but I'm at a loss for how to make this direct the workflow to either proceed to the next step or fail the entire package if any the others haven't been run (desired result).
I really appreciate any help in this. Thank you!
In SSIS:
You can set conditions in the Control Flow by double-clicking on the arrow that links one task to another. If you set up an Execute SQL Task to run a query to check whether a package has run and then return a result into a variable, you can then set up a condition to check that variable, and only continue to the next task if it is (or isn't) a certain value.
Here's an example of how you might set up a precedence constraint so that the next step would only happen if the previous one was successful, and also a particular variable did not have a value of C:
In SQL Server Agent:
You could also - if you want to avoid running an entire package - set something up in the SQL Server Agent job instead. You could add an initial step which checks on the status of a package, then quits the job reporting success. This does mean that if the step fails for another reason (say the database it's trying to run the select statement on is down), it will still quit and report success, so be careful - you might want to set up some other specific steps first which check for such situations.
Here's an article I used a while back when I wanted to understand how to set up a SQL Server Agent job in this way:
http://sqlactions.com/2012/08/05/how-to-create-custom-schedule-for-sql-server-agent-job/
Related
I have a problem with multiple executions of the same SSIS package.
I would like to allow parallel executions each of which handles a subset of data.
So far, I am thinking of using some state variable, but I don't know where to store it.
One option is to use keep the connection open and use temp tables to coordinate the task load. However, temptables cause lots of compilation issues, and they are not maintable.
Are there any other ways to identify the current execution id of the package or scope of execution? I have found no state (either in memory or stored elsewhere) in SSIS so far which I can use to partition/isolate each execution.
So based on my comments above you can try this. I dont know it is quite what your looking for, but maybe it can give you a hint to get further.
I am calling the example with workflowid 1. This is what i mean you can change in your SQL Job agent steps, and then change the parameter on each step, so fx you could add 2 steps executing workflowid 1 and workflowid 4. Then it will only run that sequence container where the constraint is success.
Create a package variable
Create your package flow
Edit your SQL Task Get WorkflowID
Add Parametermapping to your package variable
Get the resultset into local variable called WorkflowIDrun
Make your precedence constraints so it only allows one id to pass through
Notice: You could add parentworkflowid's so that you can diverse your flow inside the sequence container if you need some of the same logic
End result when package is run with workflowid 1
Create a new SQL Job in your agent. Add the needed steps Notice; I Created two steps for workflowid 1 and 2. Truncate and delete
I then edit my step and correct the variable with the right value. This will be workflowid 1 for truncate and workflowid 2 for delete
This could of cause also be in another job you do it, that depends on your needs.
I have a Data Flow Task and then there are two Execute Sql Tasks which follow the DFT. I want the Execute Sql Tasks to still run even if there is an error in the DFT.
I am aware I can do this by redirecting the erroneous records to some log file and set the option to Ignore Failure. But I'd like to perform this without the need for an error log.
I have set the MaximumErrorCount of the DFT to a considerable large value but this only helped in getting me the parent green but the Sql Tasks did not run.
I have also tried setting the OnError event for this DFT such that the value of Propagate variable is set to False.
But still no luck.
Any ideas/suggestions would be highly appreciated.
Thanks!
As you alluded to in your comment, you need to double click the connector line from the data flow to your execute sql task and change the precedence constraint from success to completion.
I am running a SSIS package using SQL Server 2008 Job. The package crash at some point while running. I have created my own mechanism to grab the error and record it in a table. So I can see that there is an error with an specific task, but could not find what the error is.
When I run the same package from BIDS, it works perfect. no error.
What I want to do is, I need to write that error string to my own table which shown in the "Execution Result" tab.
So the question is which system variable holds the error string in SSIS.
The error is stored in the ErrorDescription system variable. See Handling Errors in the Data Flow for an example of how to get the error description.
Also, if you want to capture error information into a table, SSIS supports logging to a table using the SQL Server Log Provider. You can also customize the logging.
Too easy.
Left-Click (highlight) on the object you want to capture the error event (Script, or Data Flow, etc.)
Click on 'Event Handlers' - screen should open with Executable = object you clicked and Event Handler = OnError
Click URL (click here to create....)
Drag Execute SQL object from SSIS Toolbox
Configure to the database/table you want to house the error message
Write INSERT INTO DB.Schema.Table(DBName, SchemaName, TableName,ErrorMessage,DateAdded)
Write VALUES (?,?,?,'I am smart',getdate())
Click Parameters and select the USER::Variables for the ?'s + my comment.
Since this is ran at the database server it will pass in the ?'s. My SAC is already at the database as a value but you will have selected System::ErrorDescription as parameter 3. Remember, this array is 0 based. DO NOT TRY TO NAME THE PARAMETERS. Instead, number them 0 to ~? The datatypes are based on what you have going in; mine are all VARCHAR so... :)
This is a much better solution than just logging whatever the server allows you to.
I can also add a counter variable and adjust it wherever I like; then pass it to the event OnError. This will allow me to pinpoint exactly where the last successful object completed; works best in scripting objects but also available in other areas.
I'm using this so I can process thousands of cycles without actually failing the package. If a table doesn't exist or a column doesn't exist I simply log it for further review later. Oh yeah, I'm cycling through hundreds of databases capturing their architecture and maximum column size used; not to be confused with maximum column size.
Example: TelephoneNumber comes from a source column of char(500) (definitely bad programming but...you can't change everything so..). I capture the max len of that column and adjust the destination column to accommodate that size +/- a certain percentage.
If a table doesn't exist or a column doesn't exist anymore I log the error and keep churning. At the end, I can evaluate those entries and see if I can actually remove them from my warehouse. This happens more in the TEST and STAGE environments than in PROD. However, when a change goes through to PROD I most definitely will identify it as it's coming in to the warehouse.
Everything is configured, this includes dynamic MERGE/JOINs, INSERT, SELECT, ELEMENTS, SIZES, USAGESIZE, IDENTITY, SOURCEORDER, etc. with conversions of data to destination datatypes.
ALL that because the systemic version of logging will not provide you with the granularity you might need for this type of operation. This OnError Event Handler can if setup properly.
Check this out! He has explained with a Step by step process on how to configure SSIS logging which has the error message parameter.
I am in process of creating an ssis package that need to do following in specified order:
process some data
move that data to some other tables
Get some data and push it in a plain text file.
I have created 3 store procedure for these, I have 2 "Execute SQL tasks" for 1 and 2 and a "Data Flow task" for 3rd.
Now when i run the package i can see all 3 step are completed (no errors) but they are not running in correct order.
I see step 3 is run first then step 1 and 2, i think then step 3 runs again. Normally i can ignore it but as the data in the text file can be 700 mb, i need to find a way to get SSIS to run these task in sequence.
I have tried "Sequence Container" but no luck.
Can some one help me with this please?
KA
You need to use precedence constraints to tell SSIS what order your tasks need to be executed in.
Drag the green arrow from task one to task two, and from task two to task three.
You could connect as
first SQL execute task
precedence constraint on success
second SQL execute task
precedence constraint on success
data flow
SSIS will follow the sequence as we required.
thanks
prav
I had exactly this problem. Tasks were being executed in something like the order I'd created them rather than the sequence I specified later. It turned out that I'd managed to get a task that belonged to the first sequence container to appear in the last sequence container without loosing it's allegiance to the first. I discovered this by taking a backup and deleting sequence containers - the rogue task disappeared when I deleted the first sequence container.
The fix was to cut and past the task into the desired sequence container.
I encounterd an issue on SQL Server Denali when individual components were running out of sequence even though they were joined by success constraints. The problem seemed to occur when I had cut and pasted the components and the constraint. By deleting and reapplying the constraints, the package then ran in the correct order.
In my case, if I want to decide execute order in sequence containers, I will use [sub sequence containers] between execute sql task and data flow task. Hope useful for you.
The best is to use Sequence Containers... basically they help in creating a Sequence.
But since it does not work in your case, create Child Packages for all your different process
and then create the Master Package which will have a link to those child packages, USE "Execute Package task"
I have sql server script and i want to automatic run it on the database every week.
any help.
i tried sql server agent job but i have alot of data bases on my server and i should make step to every database and it will run in the same day and same time.
You were on the right track, in management studio:
(1) Find SQL Agent and right click and select Job.
(2) Give you job a name and description and then choose the Steps option on the left
(3) In the Step options you can name your step, insert your script code, AND select which database you want the job to be performed on.
(4) Next go to the Schedules option on the left and designate how often you want the job to run (pretty self-explanatory)
(5) The Alerts, Notifications, etc options should also be utilized if you want to be informed on failure/success/etc.
That's it pretty much.
If you want one job with one step that performs the same action on every database (or perhaps a subset of databases), consider using system procedure sp_msForEachDB. I spelled out one way to use this in this prior answer.