So two part question.
First, I recently encountered an error in a listener for a spring batch job. The job was processing 2000 records and successfully processed 1790 of them. The listener only processed about 100 records before it hit the exception and failed. There weren't any other exceptions in the job that got logged, aside from the same exception getting logged and preceded by
2018-01-10 15:21:24.798 ERROR 16416 --- [pool-5-thread-7] o.s.batch.core.job.AbstractJob : Exception encountered in afterStep callback
Is it possible that the exception that killed the listener also stopped the job before it finished processing/committing the remaining 210 records? It doesn't seem to make a lot of sense, but I'm not sure what else could've resulted in those records not being committed.
Second, was what order does spring batch read in input files? The 210 records in the above scenario were the first 210 in the file. I was wondering if that's because spring batch reads in files from the last line to the first line which would explain why those were the records that didn't get processed.
First of all, the processing logic should also be written in Tasklet, ItemReader, ItemWriter, and ItemProcessor. Listeners should only be used if you want to do something before/After the step. Listeners should be used to pass the information between the steps or if you want to save the information or call the other system which is not part of batch to inform that the step is completed.
The Spring batch source code where the call to the listener is made. If you check the source code for AbstracStep
try {
// Update the step execution to the latest known value so the
// listeners can act on it
exitStatus = exitStatus.and(stepExecution.getExitStatus());
stepExecution.setExitStatus(exitStatus);
exitStatus = exitStatus.and(getCompositeListener().afterStep(stepExecution));
}
catch (Exception e) {
logger.error(String.format("Exception in afterStep callback in step %s in job %s", name, stepExecution.getJobExecution().getJobInstance().getJobName()), e);
}
It is just catching the exception and moving on. Not sure, why it has been handled this way.
If you still want the spring batch to abort if something fails in the listener, all you need to do is catch the Exception in the listener and write the below code
public class YourListener implements StepExecutionListener {
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
try {
yourcode ///////
} catch (Exception e) {
//log your error
stepExecution.setStatus(BatchStatus.FAILED);
stepExecution.addFailureException(e);
return ExistStatus.FAILED.addDescription(e);
}
return ExitStatus.COMPLETED;
}
}
I know this is something weird/wrong way to do it. But if you don't catch your exception, Spring batch will do it for you and just log. It will move ahead without aborting/stopping the batch execution.
Related
Suppose I have a supplier and some exception occurs while invoking get method of that supplier.
Supplier<> supplier = () -> getSomething();
ResilienceDecorator.executeSupplier( //Edit 1 - Could be ResilienceDecorator.executeCallable
supplier,
resilienceConfiguration,
throwable -> {
log.error("Exception occured", throwable);
return null;
});
Edit 1 - Also same for ResilienceDecorator.executeCallable.
I need a consistent API to know what went wrong during execution i.e. What was the checked exception (Edit 1 - or Unchecked exception) causing the failure so I can
handle business logic. The throwable is not root cause of exception or whatever the supplier had thrown.
If we do not provide a throwable function like above then
every exception is wrapped in a ResilienceRuntimeException and we need a chain of getCause().getCause() to know what went wrong. This is internal to sdk which might change. Again a consistent API is needed.
Is there any alternative & consistent way to know what exception the supplier is throwing? I might need to bubble it up or write a user friendly message based on the exception occured, depending upon buisness logic.
Edit 1-
An example of my current situation.
Supplier<MatDocItm> supplier = () -> {
ErpHttpDestination erpHttpDestination = DestinationAccessor.getDestination(S4_SYSTEM).asHttp().decorate(DefaultErpHttpDestination::new);
return new CustomCommand(erpHttpDestination, params).execute();
}
try {
MatDocItm = ResilienceDecorator.executeSupplier(
supplier,
configuration
);
} catch (ResilienceRuntimeException e) {
throw new ReadException("Failed to read from SAP S/4HANA", e);
}
The supplier can throw runtime exceptions and based on these exceptions I want to give a User friendly error message with proper description.
The supplier can throw a DestinationAccessException (which is runtime exception) if a destination cannot be accessed, is not configured correctly, or does not fulfill certain prerequisites.
Not only the exceptions occured in supplier, I have TimeLimiterConfiguration & if timeout occurs then a TimeoutException can be thrown.
For above MatDocItm example the ResilienceRuntimeException.getCause() will give com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException. ResilienceRuntimeException.getCause().getCause() will reveal exception that actually caused the error i.e DestinationAccessException. Now ThreadContextExecutionException is internal to sdk and can change if sdk implementation changes. Will this implementation detail remain frozen and never change in future or is there any alternative way to get the root cause?
I would recommend ExceptionUtils.throwableOfType for matching exception type and sub classes.
Alternatively ExceptionUtils.throwableOfThrowable for matching exact type only.
The utility class already provided by dependency org.apache.commons:commons-lang3:3.10
Example:
#Nullable
YourException cause = ExceptionUtils.throwableOfType(throwable, YourException.class);
You can handle the nullable result in an If statement.
As a fan of fluent APIs, I would rather go with java.util.Optional or io.vavr.control.Option.
I have a Camel route in which I'm trying to process a file and then try to insert a row to a database based on whether the file processing was successful. I just add a property "processed" and set it to true if the processing was successful in the bean I called. Since I'm checking these conditions I have nested choice and otherwise statements and I just wanted to make sure my statements are properly written. This is the basic structure of my route:
from("file:/test")
.choice()
.when(header("CamelFileName").endsWith(".txt"))
.bean(new ProcessFileBean())
.choice()
.when(exchangeProperty("processed").isEqualTo(true))
.to("sql: insert into table (id, file) values (1, file)")
.otherwise()
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
logger.info("File has not been processed");
}
});
Also, can I catch any exceptions from the route, for example during the database insert process and log them to a file?
Yes the route should work using the route you have defined.
To catch exceptions use the OnException syntax to take some action based on a particular exception. For example OnException(IOException.class).handled(true).to("error");
You can also catch exceptions similar to the try catch in normal java. See here:
http://camel.apache.org/try-catch-finally.html
I have encountered strange behavior of Java 8 CompletableFuture.exceptionally method. If I execute this code, it works fine and prints java.lang.RuntimeException
CompletableFuture<String> future = new CompletableFuture<>();
future.completeExceptionally(new RuntimeException());
future.exceptionally(e -> {
System.out.println(e.getClass());
return null;
});
But if I add another step in the future processing like thenApply, the exception type changes to java.util.concurrent.CompletionException with the original exception wrapped inside.
CompletableFuture<String> future = new CompletableFuture<>();
future.completeExceptionally(new RuntimeException());
future.thenApply(v-> v).exceptionally(e -> {
System.out.println(e);
return null;
});
Is there any reason why this should be happening? In my opinion, it's quite surprising.
This behavior is specified in the class documentation of CompletionStage (fourth bullet):
Method handle additionally allows the stage to compute a replacement result that may enable further processing by other dependent stages. In all other cases, if a stage's computation terminates abruptly with an (unchecked) exception or error, then all dependent stages requiring its completion complete exceptionally as well, with a CompletionException holding the exception as its cause.
It’s not that surprising if you consider that you may want to know whether the stage you have invoked exceptionally on failed, or one of its direct or indirect prerequisites.
yes, the behavior is expected, but if you want the original exception which was thrown from one of the previous stages, you can simply use this
CompletableFuture<String> future = new CompletableFuture<>();
future.completeExceptionally(new RuntimeException());
future.thenApply(v-> v).exceptionally(e -> {
System.out.println(e.getCause()); // returns a throwable back
return null;
});
Code
double timeout_in_hours = 6.0;
MyDataContext db = new MyDataContext();
using (TransactionScope tran = new TransactionScope( TransactionScopeOption.Required, new TransactionOptions(){ IsolationLevel= System.Transactions.IsolationLevel.ReadCommitted, Timeout=TimeSpan.FromHours( timeout_in_hours )}, EnterpriseServicesInteropOption.Automatic ))
{
int total_records_processed = 0;
foreach (DataRow datarow in data.Rows)
{
//Code runs some commands on the DataContext (db),
//possibly reading/writing records and calling db.SubmitChanges
total_records_processed++;
try
{
db.SubmitChanges();
}
catch (Exception err)
{
MessageBox.Show( err.Message );
}
}
tran.Complete();
return total_records_processed;
}
While the above code is running, it successfully completes 6 or 7 hundred loop iterations. However, after 10 to 20 minutes, the catch block above catches the following error:
{"The transaction associated with the current connection has completed but has not been disposed. The transaction must be disposed before the connection can be used to execute SQL statements."}
The tran.Complete call is never made, so why is it saying the transaction associated with the connection is completed?
Why, after successfully submitting hundreds of changes, does the connection associated with the DataContext suddenly enter a closed state? (That's the other error I sometimes get here).
When profiling SQL Server, there are just a lot of consecutive selects and inserts with really nothing else while its running. The very last thing the profiler catches is a sudden "Audit Logout", which I'm not sure if that's the cause of the problem or a side-effect of it.
Wow, the max timeout is limited by machine.config: http://forums.asp.net/t/1587009.aspx/1
"OK, we resolved this issue. apparently the .net 4.0 framework doesn't
allow you to set your transactionscope timeouts in the code as we have
done in the past. we had to make the machine.config changes by adding
< system.transactions> < machineSettings maxTimeout="02:00:00"/>
< defaultSettings timeout="02:00:00"/> < /system.transactions>
to the machine.config file. using the 2.0 framework we did not have
to make these entries as our code was overriding teh default value to
begin with."
It seems that the timeout you set in TransactionScope's constructor is ignored or defeated by a maximum timeout setting in the machine.config file. There is no mention of this in the documentation for the TransactionScope's constructor that accepts a time out parameter: http://msdn.microsoft.com/en-us/library/9wykw3s2.aspx
This makes me wonder, what if this was a shared hosting environment I was dealing with, where I could not access the machine.config file? There's really no way to break up the transaction, since it involves creating data in multiple tables with relationships and identity columns whose values are auto-incremented. What a poor design decision. If this was meant to protect servers with shared hosting, it's pointless, because such a long-running transaction would be isolated to my own database only. Also, if a program specifies a longer timeout, then it obviously expects a transaction to take a longer amount of time, so it should be allowed. This limitation is just a pointless handicap IMO that's going to cause problems. See also: TransactionScope maximumTimeout
This is a refactoring question.
try
{
string line = GetFirstLineFromFile(); //Gets first line from a text file, this line would be a number.
int value = ConvertToInteger(line); // Gets the integer value from the string.
int result = DivideByValue(value); // Divides some number with the value retrieved.
}
catch(Exception ex)
{
}
My main concern is, what is the best approach for exception handling in such situations. Certainly wrapping the whole thing in a single try catch is like saying I expect an exception about everything. There must be some place we catch a generic exception right?
Just don't catch a "generic exception".
How can you possibly handle ANY exception and know how to keep your application in a clean state ?
It hides bugs and it's a really bad idea.
Read this serie of posts on catch (Exception).
You need to think about what exceptions can be thrown from the methods in the try block, as well as which ones of those you can deal with at the current level of abstraction.
In your case, I'd expect that the getFirstLineFromFile methods, for example, can definitely throw exceptions you'd want to catch here. Now whether you wrap and rethrow the exception, or take other action, depends on whether you can actually deal with the exception at this level. Consider the case where you have a default file you can fall back to - the approach may just be to log a warning and continue with the default. Or if the whole application is based on reading a file supplied by the user, then this is more likely to be a fatal exception that should be propagated up to the top level and communicated to the user there.
There's no hard-and-fast rule like "always throw" or "never throw"; in general, I consider that one should throw exceptions whenever there's an exceptional-type situation that is not considered a normal result of the method, and consequently cannot be adequately described by the return type of the method. (For example, an isValidDbUser method returning boolean might be able to handle SQLExceptions as just return false; but a getNumProductsRegisteredInDB returning an int should almost certainly propagate an exception).
Don't listen to the (hordes) of people that will tell you that you should never catch multiple exceptions in one big general block. It's a perfectly reasonable way to do general error handling in some cases, which is why the ability to do so exists in the language.
There will be some exceptions that you can do something specific and useful about (i.e. recover from them in the catch block.) These are the kinds of exceptions that you want to catch individually, and as close to the place where they occur as possible.
But the majority of exceptions that you'll face in real life will be completely unexpected, unchecked exceptions. They are the result of programmer error (bugs), failed assertions, failing hardware, dropped network connections, etc.
You should design your software defensively, by designating specific "chokepoints" to handle these unpredictable exceptions with a minimum of disruption to the rest of the application. (Remember, in many cases, "handling" the exception often just means aborting the current operation and logging an error or otherwise telling the user that an unexpected error occurred.)
So for example, if your program saves a file to the disk, you could wrap the entire save operation in a try block to catch things that goes wrong during the save:
try {
// attempt to perform the save operation
doSave();
} catch (Throwable t) {
// tell the user that the save failed for unexpected reasons
// and log the error somewhere
informUser("save failed!");
log("save failed!", t);
} finally {
// last minute cleanup (happens whether save succeeded or failed)
...
}
Notice that we choose a nice chokepoint method here ( doSave() ) and then stop any unexpected errors from bubbling up any further than this point. The chokepoint represents a single, cancellable operation (a save). While that operation is obviously toast if you're getting an unexpected exception, the rest of the application will remain in a good state regardless of what happens on the other side of the chokepoint.
Also notice that this idiom does NOT stop you from handling some of your exceptions further down in doSave() somewhere. So if there are exceptions that might get thrown that you can recover from, or that you want to handle in a special way, go ahead an do so down in doSave(). But for everything else, you have your chokepoint.
You might even want to set up a general uncaught exception handler for your entire program in your main method:
public static void main(String [] args) {
try {
startApplication();
} catch (Throwable t) {
informUser("unexpected error! quitting application");
log("fatal application error", t);
}
But if you've set your other chokepoints up wisely, no exceptions will ever bubble up this far. If you want to make your general error handling complete, you can also create and assign an UncaughtExceptionHandler to important threads, including your main thread or the AWT thread if you are using Swing.
TL;DR; Don't believe the dogma that you should always catch exceptions as specifically as possible. There are times when you want to catch and handle a specific exception, and other times when you want to use chokepoints to catch and deal with "anything else that might go wrong".