This is a follow up to the previous question (requirements given in original question).
Spring Integration - Filter - Send messages to a different end point
My issue is that if there is more than one error in the input file, only the first error is getting logged. The subsequent errors are not getting logged.
Modified code:
#Configuration
public class CreateUserConfiguration {
#Bean
public IntegrationFlow createUser() {
return IntegrationFlows.from(Files.inboundAdapter(new File(INPUT_DIR)))
.enrichHeaders(h -> h.header("errorChannel", "exceptionChannel", true))
.transform(csvToUserBeanTransformer, "convertCsvToUserBean")
.split(userBeanSplitter, "splitUserBeans")
.wireTap(flow -> flow.<UserBean>filter(userBean -> !userBean.getStatus().equalsIgnoreCase("SUCCESS")).channel("errorSummaryReportGenerationChannel"))
.transform(userBeanToJSONTransformer, "convertUserBeanToJSON")
.handle(Files.outboundAdapter(new File(OUTPUT_SUCCESS_DIRECTORY)))
.get();
}
#Bean
public IntegrationFlow logErrorSummary() {
return IntegrationFlows.from("errorSummaryReportGenerationChannel")
.handle((p,h) -> {
return ((UserBean)(p)).getUserID() + "\t" + ((UserBean)(p)).getStatus();
})
.transform(Transformers.objectToString())
.handle(Files.outboundAdapter(new File(OUTPUT_FAILED_REPORT_FILE_NAME)))
.get();
}
#Bean
public IntegrationFlow logError() {
return IntegrationFlows.from("exceptionChannel")
.enrichHeaders(h -> h.headerExpression("errorFileName", "payload.failedMessage.headers.fileName"))
.wireTap(flow -> flow.handle(msg -> System.out.println("Received on exceptionChannel " + msg.getHeaders().get("errorFileName"))))
.transform(Transformers.objectToString())
.handle(Files.outboundAdapter(new File(generateOutputDirectory(OUTPUT_FAILED_DIRECTORY))).autoCreateDirectory(true).fileExistsMode(FileExistsMode.APPEND).fileNameExpression("getHeaders().get(\"errorFileName\")+'.json'"))
.get();
}
#Bean(name = "exceptionChannel")
MessageChannel exceptionChannel() {
return MessageChannels.executor(new SimpleAsyncTaskExecutor()).get();
}
#Bean(name="errorSummaryReportGenerationChannel")
MessageChannel errorSummaryReportGenerationChannel() {
return DirectChannel();
}
}
WHAT I EXPECT:
In errorSummaryReport -
B123 ERROR, FREQUENCY
C123 FREQUENCY_DETAIL
In OUTPUT_FAILED_DIRECTORY -
B123.json -> stacktrace of error
C123.json -> stacktrace of error
WHAT I SEE: (C123 information is missing)
In errorSummaryReport -
B123 ERROR, FREQUENCY
In OUTPUT_FAILED_DIRECTORY -
B123.json -> stacktrace of error
The problem pops up from the .split(userBeanSplitter, "splitUserBeans").
I would say it is fully similar to what we do in plain Java with a for loop.
So, if method in the loop throws an exception, you indeed go away from the loop and the next item is not going to be processed any more.
To fix your problem you need to add a .channel(c -> c.executor(myExecutor())) to process splitted items in parallel and have a error handling in the separate thread. This way the loop in the splitter is not going to be affected.
Related
So, by reading the NestJS documentation, I get the main idea behind how the filters work with exceptions.
But from all the code I have seen, it seems like all services always throw HttpExceptions.
My question is: Should the services really be throwing HttpExceptions? I mean, shouldn't they be more generic? And, if so, what kind of Error/Exception should I throw and how should I implement the filter to catch it, so I won't need to change it later when my service is not invoked by a Http controller?
Thanks :)
No they should not. An HttpException should be thrown from within a controller. So yes, your services should expose their own errors in a more generic way.
But "exposing errors" doesn't have to mean "throwing exceptions".
Let's say you have the following project structure :
📁 sample
|_ 📄 sample.controller.ts
|_ 📄 sample.service.ts
When calling one of your SampleService methods, you want your SampleController to know whether or not it should throw an HttpException.
This is where your SampleService comes into play. It is not going to throw anything but it's rather going to return a specific object that will tell your controller what to do.
Consider the two following classes :
export class Error {
constructor(
readonly code: number,
readonly message: string,
) {}
}
export class Result<T> {
constructor(readonly data: T) {}
}
Now take a look at this random SampleService class and how it makes use of them :
#Injectable()
export class SampleService {
isOddCheck(numberToCheck: number): Error | Result<boolean> {
const isOdd = numberToCheck%2 === 0;
if (isOdd) {
return new Result(isOdd);
}
return new Error(
400,
`Number ${numberToCheck} is even.`
);
}
}
Finally this is how your SampleController should look like :
#Controller()
export class SampleController {
constructor(
private readonly sampleService: SampleService
) {}
#Get()
sampleGetResponse(): boolean {
const result = this.sampleService.isOddCheck(13);
if (result instanceof Result) {
return result.data;
}
throw new HttpException(
result.message,
result.code,
);
}
}
As you can see nothing gets thrown from your service. It only exposes whether or not an error has occurred. Only your controller gets the responsibility to throw an HttpException when it needs to.
Also notice that I didn't use any exception filter. I didn't have to. But I hope this helps.
Could you please give me an advice, what is the proper way to return HttpStatus.BAD_REQUEST from webflux handler?
Here's the code, it seems to work properly, exception message returns as needed, but the response status is 200, while needed to be 400. Please note, that returning media type is TEXT_EVENT_STREAM -
#Component
public class ResolveHandler {
public Mono<ServerResponse> resolve(ServerRequest request) {
return ServerResponse.ok().contentType(MediaType.TEXT_EVENT_STREAM)
.body(
new ResultsFluxer(
Integer.valueOf(request.queryParam("numberOfIterations").orElse("0")),
request.queryParam("function1").orElse(""),
request.queryParam("function2").orElse(""),
orderingType
)
.getResult()
.onErrorResume(e -> Flux.just(e.getMessage()))
, Flux.class);
}
}
I have a Spring Batch Task running on our cloud platform that will launch with the provided command line parameters, and then skip over the execution of the first Step with the following error:
[OUT] The job execution id 992 was run within the task execution 1325
[OUT] Step already complete or not restartable, so no action to execute:
StepExecution: id=1071, version=3, name=OFileStep, status=COMPLETED, exitStatus=COMPLETED, readCount=0, filterCount=0, writeCount=0 readSkipCount=0,
writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0, exitDescription=
I have investigated the metadata tables in the MySQL instance that Spring Batch uses to find that the JOB_INSTANCE_ID is the same between multiple executions, when it should increment by 1 each time.
The #Bean that I have defined for the Job Configuration is:
#Bean
public Job job() {
return jobBuilderFactory.get(OTaskConstants.JOB_NAME)
.listener(listener())
.incrementer(new RunIdIncrementer())
.start(dataTransferTaskStep())
.next(controlMTaskStep())
.build();
}
Is anyone aware of what could be causing this behavior?
Below line clearly says it all.
Step already complete or not restartable, so no action to execute:
Meaning the step/job already complete and can not be restarted. This is the behavior of Spring Batch. In order to by pass this we need to pass an unique argument.
In your case i see you already have RunIdIncrementer. Now question is why it is not working.
Can you see BATCH_JOB_PARMS table to see what arguments are getting passed to the job? May be you are missing something.
You can also use SimpleIncrementor. See below code for explanation.
https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#JobParametersIncrementer
Remove #Bean annotation on Job.
It causes the Job to be launched with no parameters every time you launch/start application as spring tries to load the bean and which in-turn launches the batch job.
Remove the annotation and use spring scheduler to schedule the jobs.
I had the same issue. Below code helped me resolve it. By adding params in job launcher a new job_instance_id is created for every run.
#SpringBootApplication
public class App implements CommandLineRunner {
#Autowired
JobLauncher jobLauncher;
#Autowired
Job job;
public static void main(String[] args) {
SpringApplication.run(App.class, args);
}
#Override
public void run(String... args) throws Exception {
JobParameters params = new JobParametersBuilder()
.addString("JobID", String.valueOf(System.currentTimeMillis()))
.toJobParameters();
jobLauncher.run(job, params);
}
}
Solution
Refer error message above “If you want to run this job again, change the parameters.” The formula is JobInstance = JobParameters + Job. If you do not have any parameters for JobParameters, just pass a current time as parameter to create a new JobInstance. For example,
CustomJobLauncher.java
//...
#Component
public class CustomJobLauncher {
#Autowired
JobLauncher jobLauncher;
#Autowired
Job job;
public void run() {
try {
JobParameters jobParameters =
new JobParametersBuilder()
.addLong("time",System.currentTimeMillis()).toJobParameters();
JobExecution execution = jobLauncher.run(job, jobParameters);
System.out.println("Exit Status : " + execution.getStatus());
} catch (Exception e) {
e.printStackTrace();
}
}
}
Source : https://mkyong.com/spring-batch/spring-batch-a-job-instance-already-exists-and-is-complete-for-parameters/
Preface: I'm fairly new to Camel, and after digesting Camel in action as best as possible, I'm adapting it to a project I'm on. In this project, we have some rather complex error handling, and I want to make sure I can replicate this as we Camel-ize our code.
On our project (as most) there are a set of Exceptions we want to retry and a set that we don't - but more specifically, there are a set that we want to retry more than others (not all recoverable errors can be treated the same). In this case, I was attempting to define an onException block to change the redelivery policy. However, it seems that the Exchange maintains the count (Exchange.REDELIVERY_COUNTER) and that this count is not dependent on which exception is thrown. Is there a way to make this count be specific for a given exception?
For example - I have two exceptions FooException and BarException. In my route (or really in the whole context), I want to retry FooExceptions 10 times, but BarExceptions should only retry 2 times. So the context will contain:
<onException>
<exception>my.exception.FooException</exception>
<redeliveryPolicy maximumRedeliveries="10" redeliveryDelay="2000"
</onException>
<onException>
<exception>my.exception.BarException</exception>
<redeliveryPolicy maximumRedeliveries="2" redeliveryDelay="5000"
</onException>
Now, the concern - if my application throws a FooException and retries 4 times (each time throwing a FooException) and then on the 5th attempt, it throws a BarException, it seems that the way this works is the Exchange will have a REDELIVERY_COUNTER of 5, and when I reset the policy to only try twice, it (logically) concludes that the route should not be retried and throws the exception back out. However, in my application BarExceptions should be retried twice, regardless of how many FooExceptions get thrown. And likewise, if it alternates throwing Foo and Bar exceptions, I would like it to only increment the counter for the given exception.
The very end of Camel in Action promotes using a retryWhile - is this the only way to grab the kind of control I'm looking for? Do I need to create a stateful bean that is aware of the count per exception? Or am I overlooking something simple? I want to make sure that as I approach this refactor I don't start us off on an ugly path.
Using Camel 2.10.1
I checked your case with following test:
import org.apache.camel.EndpointInject;
import org.apache.camel.Exchange;
import org.apache.camel.Produce;
import org.apache.camel.ProducerTemplate;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.mock.MockEndpoint;
import org.apache.camel.test.junit4.CamelTestSupport;
import org.junit.Test;
import java.util.concurrent.atomic.AtomicLong;
/**
* #author Illarion Kovalchuk
* Date: 12/7/12
* Time: 2:58 PM
*/
public class Test extends CamelTestSupport
{
private static final String MIDDLE_QUEUE = "seda:middle";
#EndpointInject(uri = "mock:result")
protected MockEndpoint resultEndpoint;
#Produce(uri = "direct:start")
protected ProducerTemplate template;
private Processor processor = new Processor();
#Test
public void shouldRedeliverOnErrors() throws Exception
{
resultEndpoint.expectedBodiesReceived("Body");
template.sendBodyAndHeader(MIDDLE_QUEUE, "Body", "Header", "HV");
resultEndpoint.assertIsNotSatisfied();
}
#Override
protected RouteBuilder createRouteBuilder()
{
return new RouteBuilder()
{
#Override
public void configure() throws Exception
{
onException(FooException.class)
.redeliveryDelay(2000)
.maximumRedeliveries(10);
onException(BarException.class)
.redeliveryDelay(5000)
.maximumRedeliveries(2);
from(MIDDLE_QUEUE)
.bean(Processor.class, "process")
.to(resultEndpoint)
.end();
}
};
}
public static class Processor
{
private static AtomicLong retryState = new AtomicLong(0L);
public static void process(Exchange e) throws FooException, BarException
{
long rs = retryState.getAndAdd(1L);
if (rs < 4)
{
System.err.println("Foo Attempt "+ rs);
throw new FooException();
}
if (rs == 4)
{
System.err.println("Bar Attempt "+ rs);
throw new BarException();
}
System.err.println("Normal Attempt "+ rs);
}
}
public static class FooException extends Throwable
{
}
private static class BarException extends Throwable
{
}
}
As the result, your concirn was approved: delivery attempts gets exhausted after BarException, even if we have only 4 FooExceptions and 1 BarException.
Unfortunately I can't answer your question fully right now, but I am digging into it and will updated my unswer if get something new.
Try to replace the order you define your exceptions, e.g.:
<onException>
<exception>my.exception.BarException</exception>
<redeliveryPolicy maximumRedeliveries="2" redeliveryDelay="5000"
</onException>
<onException>
<exception>my.exception.FooException</exception>
<redeliveryPolicy maximumRedeliveries="10" redeliveryDelay="2000"
</onException>
I am running into an extremely strange behavior in Groovy. When I throw an exception from a closure in a Script, the end exception that was thrown was different.
Here are the code and the details:
public class TestDelegate {
def method(Closure closure) {
closure.setResolveStrategy(Closure.DELEGATE_FIRST);
closure.delegate = this;
closure.call();
}
public static void main(String[] args) {
// Make Script from File
File dslFile = new File("src/Script.dsl");
GroovyShell shell = new GroovyShell();
Script dslScript = shell.parse(dslFile);
TestDelegate myDelegate = new TestDelegate();
dslScript.metaClass.methodMissing = {
// will run method(closure)
String name, arguments ->
myDelegate.invokeMethod(name, arguments);
}
dslScript.metaClass.propertyMissing = {
String name ->
println "Will throw error now!"
throw new MyOwnException("errrrror");
}
dslScript.run();
}
}
class MyOwnException extends Exception {
public MyOwnException(String message) {
super(message);
}
}
Script.dsl:
method {
println a;
}
So the plan is that when I run the main() method in TestDelegate, it will run the DSL script, which calls for the method method(). Not finding it in the script, it will invoke methodMissing, which then invokes method() from myDelegate, which in turns invoke the closure, setting the delegate to the testDelegate. So far, so good. Then the closure is supposed to try printing out "a", which is not defined and will thus set off propertyMissing, which will will throw MyOwnException.
When I run the code, however, I get the following output:
Will throw error now!
Exception in thread "main" groovy.lang.MissingPropertyException: No such property: a for class: TestDelegate
Now, it must have reached that catch block, since it printed "Will throw error now!" It must have thrown MyOwnException too! But somewhere along the lines, MyOwnException was converted to MissingPropertyException, and I have no idea why. Does anyone have any idea?
P.S. if I remove closure.setResolveStrategy(Closure.DELEGATE_FIRST) from TestDelegate#method(), the code acts as expected and throws MyOwnException. But I really need the setResolveStrategy(Closure.DELEGATE_FIRST) for my DSL project. And I would prefer to know the root cause of this rather than just removing a line or two and see that it works without understanding why.
I think this is what essentially happens: With a delegate-first resolve strategy, the Groovy runtime first tries to access property a on myDelegate, which results in a MissingPropertyException because no such property exists. Then it tries propertyMissing, which causes a MyOwnException to be thrown. Eventually the runtime gives up and rethrows the first exception encountered (a design decision), which happens to be the MissingPropertyException.
With an owner-first resolve strategy, propertyMissing is consulted first, and hence MyOwnException is eventually rethrown.
Looking at the stack trace and source code underneath should provide more evidence.