I have attached the screen shot
How to use the check points in SSIS 2012 package in order to restart the package from point of failure not from the beginning.
I think its not allowing me to attach the image...
You can use Checkpoints which basically executes the package from the point where it got failed.If your package has 5 to 6 steps and if the 2nd step failed then the next time when u execute the package it will start from 2nd step .
You can refer these blogs 31 days of SSIS and articles from Simple Talk
Update :-
Step1: Right click on Control Flow and select properties.
Step2: Specify the informations for CheckPointFileName, CheckpointUsage and SaveCheckPoints
CheckPointFileName = Points to the XML file location where checkpoint details will be stored by SSIS
CheckpointUsage =Specifies 3 properties (Never,IfExists,Always)
Never signifies that the checkpoints will never be used.
IfExists = Checkpoints will be used if a file exists
Always=A checkpoint file will always be used .If the file is missing then error will be thrown
SavecheckPoints =Set it true to save the checkpoints
Now for all the controls or tasks set FailPackageOnFailure property to True for the components which you want to participate in Checkpoints . This property indicates whether the package fails when the execution fails
Related
I have an SSIS package where the checkpoints are not behaving as I understand that they should.
To simplify, this is the kind of setup:
Imagine a package with two containers in a serial flow (Container 1 executes then Container 2). Checkpoints are configured and ONLY Container 2 is setup as "Fail Package on Failure". It is desired that Container 1 is ALWAYS completely re-run even in a job restart unless the job made it to Container 2, then it should restart at a checkpoint in Container 2.
Container 1 has a major table copy step that obviously can't be restarted using checkpoints since checkpoints do not track db updates. So Container 1 has logic that allows it to restart the "old fashion way" by interrogating the source and destination, determining where it left off when it failed, and updating package variables accordingly that are used to drive the main query so it will resume at the row it left off. This works flawlessly if I delete the checkpoint file after the failure and restart.
I intentionally cause a failure during the table copy in the non-checkpointed Container 1. But with the checkpoint file left at failure, it doesn't work! For some reason upon restart, it acts as if Container 1 is also checkpointed. It does NOT run the steps in Container 1 that will interrogate the source and destination, nor the steps that update the variables and instead immediately goes to the step of the table copy in Container 1, using variable values from the failed run, which causes the copy to start at an incorrect position (and end up with pk violations in insert). Every task in Container 1 has "Fail Package on Failure" set to FALSE. They also have "Fail Parent on Failure" set to false in case that matters. Why does it act as if Container 1 is checkpointed? Why doesn't it run all steps in Container 1 from the beginning since the package has not even gotten to the first container that is checkpointed (Container 2)? What behavior am I missing with these checkpoints?
I have created a child SSIS package that executes according to the "ProcessName" variable value that is specified initially. Now, I wish to create a parent package such that I can execute 4 child package tasks with different ProcessName values passed in to be executed in parallel. How can I maintain my child package and pass in different values to each of the 4 execute packages task such that the ProcessNames variable values are different for each of them? I am new to SSIS and would deeply appreciate if someone could advice or give a direction on how I could go about doing so.
I would see this as a pattern like the following
The "trick" here is that within each Sequence Container, SEQC, I need to define my variable that holds my parameter value. That variable needs to be scoped to the container - otherwise, there is only one SSIS variable and the 4 processes that attempt to initialize that value will be in conflict.
In the SSIS Variables menu, there is a Move Variable icon (second one listed)
Here you can see that I have ParameterValue defined in both "SEQC Opt 1a" and "SEQC Opt 1b" and they're initialized with different values.
The first step within the Sequence container is an Execute SQL Task where I pull back the intended parameter value. Maybe that is not needed in your case but it can be helpful to have a repository of run-time values. In the case of 1b, this is much more what my execution pattern looks like. I have a query that pulls back any packages to be run within the scope of this container and the starting value. e.g.
ContainerName|PackageName|StartingValue
SEQC Opt 1a |Child0.dtsx|100
SEQC Opt 1a |Child1.dtsx|200
SEQC Opt 1a |Child2.dtsx|300
SEQC Opt 1b |Child5.dtsx|600
SEQC Opt 1b |Child6.dtsx|700
SEQC Opt 1b |Child7.dtsx|800
This table pattern allows me to dynamically run packages in both parallel and in serial. Assuming Child7 and Child2 in the above set are very slow but the other 4 packages are relatively fast. The fast ones would start up, do their work and complete and the next runs. There are limits to how many parallel operations can fire at once so you can't scale infinitely across processes so a balance of serial and parallel operations makes sense.
Once you have your pattern working for one sequence container: copy, paste, rename and assuming you lookup in a table based on the task name as I show above, it's ready to go.
NOTE for everyone reading this answer: This answer is not full/complete with examples/full steps. From comment above I am posting this now so requestor can see it and get started.
This was from notes I wrote myself a long while back on how to do this for myself. I am posting it as answer as it is helpful and too large to post as comment. Plus I have not rewritten anything in what wrote for myself and what I am posting.
Currently I can not find my full code yet to post full details/steps. If/when I do I will post here, but this should be good details on what/how to do it. Plus this gives info on how to handle child package error trapping as well.
-- my notes I saved for myself posting as answer:
Steps for creating child packages:
Create any variables needed in the child package
Create the coorisponding variable name in the parent package (the name doesnot have to be the same, and maybe want to name it something to identify it as a child package variable)
Child Package:
Need to set up: Package Configurations
a. Right click on package and click Package Paramaters
b. Click the checkox to Enable Package configurations
Click Add and set the paramaters:
a. Configuration Type: Pareknt Package variable
b. Specify the configuration setting directly: Put the parent variable name in here that the child package is going to access
c. Click "Next"
d. In "Objects" window scroll down to the variable you are setting from the parent variable name you selected above and click the "Value" option under Properties for that variable name
e. Click "Next"
f. Under Configuration Name: Set a detailed name for what this variable is/does.
Error Handling (NOTE: This is not required but you wont capture the child error messages if you dont do this):
a. Go to Event Handlers Tab
b. Under drop down (on top right) select OnError
c. Add a Scrit Task
d. Pass as read only variables:
System::ErrorDescription
System::SourceName
System::PackageName
e. Copy/paste the code below into the script task uin the Main() function.
----- this is for the error handling
public void Main()
{
// build our the error message
string ErrorMessageToPassToParent = "Package Name: " + Dts.Variables["System::PackageName"].Value.ToString() + Environment.NewLine + Environment.NewLine +
"Step Failed On: " + Dts.Variables["System::SourceName"].Value.ToString() + Environment.NewLine + Environment.NewLine +
"Error Description: " + Dts.Variables["System::ErrorDescription"].Value.ToString();
// have to do this FIRST so you can access variable without passing it into the script task from SSIS tool box
// Populate collection of variables. This will include parent package variables.
Variables vars = null;
Dts.VariableDispenser.GetVariables(ref vars);
// checks if this variable exists in parent first, and if so then will set it to the value of the child variable
// (do this so if parent package does not have the variable it will not error out when trying to set a non-existant varaible)
if (Dts.VariableDispenser.Contains("OnError_ErrorDescription_FromChild") == true)
{
// Lock the to and from variables.
// parent variable
Dts.VariableDispenser.LockForWrite("User::OnError_ErrorDescription_FromChild");
// Need to call GetVariables again after locking them. Not sure why - perhaps to get a clean post-lock set of values.
Dts.VariableDispenser.GetVariables(ref vars);
// Set parentvar = childvar
vars["User::OnError_ErrorDescription_FromChild"].Value = ErrorMessageToPassToParent;
vars.Unlock();
}
Dts.TaskResult = (int)ScriptResults.Success;
}
Parent Package:
Add this variable to propertly capture the child error messages (not required but you wont capture chidl error messages if you dont):
variable: OnError_ErrorDescription_FromChild
Error Handling(NOTE: This is not required but you wont capture the child error messages if you dont do this):
a. Go to Event Handlers Tab
b. Under drop down (on top right) select OnError
c. Add a Scrit Task
d. Pass as read only variables:
User::OnError_ErrorDescription_FromChild
e. Copy/paste the code below into the script task uin the Main() function.
----- this is for the error handling
public void Main()
{
// get the varaible from the parent package for the error
string ErrorFromChildPackage = Dts.Variables["User::OnError_ErrorDescription_FromChild"].Value.ToString();
// do a check if the value is empty or not (so we knwo if the error came from the child package or the occurent in the parent package itself
if (ErrorFromChildPackage.Length > 0)
{
// Then raise the error that was created in the child package
Dts.Events.FireError(0, "Capture Error From Child Package Failure",
ErrorFromChildPackage
, String.Empty, 0);
//Dts.TaskResult = (int)ScriptResults.Failure;
} // end if the error length of variable is > 0
Dts.TaskResult = (int)ScriptResults.Success;
}
NOTES:
For error handling:
a. The child package error handling is written so it wont fail if the variables or error handling does not exist in parent package.
b. If you include the error handling (and variable) in the parent package it MUST exist in the child package though.
I encountered an error while doing full-import in solr-6.6.0.
I am getting exception as bellow
This happens when I set
batchSize="-1" in my db-config.xml
If I change this value to say batchSize="100" then import will run without any error.
But recommended value for this is "-1".
Any suggestion why solr throwing this exception.
By the way the data am trying to import is not huge, data am trying to import is just 250 documents.
Stack trace:
org.apache.solr.handler.dataimport.DataImportHandlerException: java.sql.SQLException: Operation not allowed after ResultSet closed
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:516)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:415)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:474)
at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:457)
at java.lang.Thread.run(Thread.java:745)
By the way am getting one more warning:
Could not read DIH properties from /configs/state/dataimport.properties :class org.apache.zookeeper.KeeperException$NoNodeException
This happens when config directory is not writable.
How can we make config directory writable in solrCloud mode.
Am using zookeeper as watch-dog. Can we go ahead and change permission of config files which are there is zookeeper?
your help greatly appreciated.
Using fetchSize="-1" is only recommended if you have problems running without it. Its behaviour is up to the JDBC driver, but the cause of people assuming its recommended is this sentence from the old wiki:
DataImportHandler is designed to stream row one-by-one. It passes a fetch size value (default: 500) to Statement#setFetchSize which some drivers do not honor. For MySQL, add batchSize property to dataSource configuration with value -1. This will pass Integer.MIN_VALUE to the driver as the fetch size and keep it from going out of memory for large tables.
Unless you're actually seeing issues with the default values, leave the setting alone and assume your JDBC driver does the correct thing (.. which it might not do with -1 as the value).
The reason for dataimport.properties having to be writable is that it writes a property for the last time the import ran to the file, so that you can perform delta updates by referencing the time of the last update in your SQL statement.
You'll have to make the directory writable for the client (solr) if you want to use this feature. My guess would be that you can ignore the warning if you're not using delta imports.
I need to elaborate the GUID of every subpackage which is executed by a main package.
In order to do that, I've handled the event Pre-Execute, to record every source of Pre-Execute event. The problem is that in PreExecute event handler, it isnt possible to know if the source of the event, represented by SourceID and SourceName, is a subpackage or a task.
I would code only in the main package, in order to centralize the code.
Other attemps:
I've tried with other events, like PreValidate, and the issue is the same.
Looking at the standard output (see below) I found a specific log row for subpackages, but I dont know how to read standard output log at runtime.
This is the standard output:
Log:
Name: User:PackageStart
Computer: COMPUTERNAME
Operator: username
Source Name: childpackage
Source GUID: {9E629F59-5DD1-44AD-BD64-9704353267E2}
Execution GUID: {4168C4C6-1EC7-41C2-8F2D-98AD3A4F4D82}
Message: Beginning of package execution.
Start Time: 2016-05-16 12:28:16
End Time: 2016-05-16 12:28:16
End Log
Thank you
I have a DTS package that calls an executable via an Execute Process Task object. The path of executable can change based on where the product that this is contained in is installed. Is there some way to make the executable path dynamic?
I tried using an expression for the executable property. I set it to the a value that came out of a stored procedure, but it seems to only calculate the value when you save the package. I tried setting DelayValidation = true, but it doesn't seem to ever update it at runtime.
I believe you have something amiss with your package. Update your question with concrete details or compare away to my sample.
Setup
I create 7 subfolders from my base location and inside each, I placed a batch file
#echo off
REM N replaced with value 0-6
ECHO C:\ssisdata\EXEC\N\RunMe.bat
This led to a structure like
C:\ssisdata\EXEC\0\RunMe.bat
C:\ssisdata\EXEC\1\RunMe.bat
C:\ssisdata\EXEC\2\RunMe.bat
C:\ssisdata\EXEC\3\RunMe.bat
C:\ssisdata\EXEC\4\RunMe.bat
C:\ssisdata\EXEC\5\RunMe.bat
C:\ssisdata\EXEC\6\RunMe.bat
When I run them, it simply reports back the hard-coded location message
SSIS
I created an SSIS package that had a For Loop Container and inside was an Execute Process Task coupled to a Script Task
Variables
FolderBase: string - C:\ssisdata\EXEC Abstracts away the common path
FolderChoice: int - 0 Montonically increasing value from 0 to 6. Use by the loop to force change the location of the executable
Output: string - `` Captures the output from the executable to prove it works as expected
CurrentExecutable: string - C:\ssisdata\EXEC\0\RunMe.bat This is an Expression based on the above variables. Expression is #[User::FolderBase] + "\\" + (DT_WSTR, 1) #[User::FolderChoice] + "\\RunMe.bat"
Execute Process Task
I did nothing of interest here. I route standard out to an SSIS Variable and I actually used C:\ssisdata\Exec\RunMe.bat as my source but the next step updated this screenshot.
On the Expressions tab, I used my Variable #[User::CurrentExecutable] and assigned it to the Executable property.
Script Task
I passed in my #[User::Output] variable and call Dts.Events.FireInformation to make the output show up.