camel read directory and choose a destination by completion task result - csv

Question

I need to read a directory with csv files (one or more files). I use Camel with springboot and I need to move any file processed completely(without errors) to OUT dir, but if the last "to" route fails (an Exceptions is thrown) I need to move the file to REFUSED dir.
When I try my code, camel goes to infinitive loop, continue to process the same file forever..
24/08/2017 16:27:57.070 ERROR [Camel (camel-1) thread #0 - file://src/main/resources/data] - org.apache.camel.processor.DefaultErrorHandler: Failed delivery for (MessageId: ID-CAD1652-39380-1503584865077-0-33 on ExchangeId: ID-CAD1652-39380-1503584865077-0-34). Exhausted after delivery attempt: 1 caught: com.cadit.exceptions.FileNotEvaluableException: Error: file tipo sconosciuto
Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[route2 ] [route2 ] [file://src/main/resources/data?idempotent=false&move=OUT%2FVB%2F ] [ 10]
[route2 ] [unmarshal1 ] [unmarshal[org.apache.camel.model.dataformat.CsvDataFormat#28f6cf0f] ] [ 1]
[route2 ] [to1 ] [bean:myCsvHandler?method=doHandleCsvDataVB ] [ 8]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
com.cadit.exceptions.FileNotEvaluableException: Error: file tipo sconosciuto
at com.cadit.handlers.MyCsvHandler.doHandleCsvDataVB(MyCsvHandler.java:172)
at com.cadit.handlers.MyCsvHandler$$FastClassBySpringCGLIB$$f4b8f70b.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.cadit.handlers.MyCsvHandler$$EnhancerBySpringCGLIB$$d81d9e7f.doHandleCsvDataVB(<generated>)
at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:458)
at org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:289)
at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:262)
at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:178)
at org.apache.camel.component.bean.BeanProducer.process(BeanProducer.java:41)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:460)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:227)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:191)
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:175)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:102)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
This is the main code:
#Component
public class CamelContextConf extends RouteBuilder{
static final Logger logger = Logger.getLogger(CamelContextConf.class);
#Override
public void configure() throws Exception {
restConfiguration().component("servlet").dataFormatProperty("prettyPrint", "true") ;
CsvDataFormat csv = new CsvDataFormat();
csv.setDelimiter(";");
csv.setSkipHeaderRecord(true);
from("direct:csvprocessor")
.streamCaching()
.from("file:src/main/resources/data?move=OUT/VB/")
.unmarshal(csv)
.to("bean:myCsvHandler?method=doHandleCsvDataVB")
.onCompletion().onFailureOnly().to("file:src/main/reources/data/REFUSED").end()
.setBody(constant("OK"))
.setHeader(Exchange.HTTP_RESPONSE_CODE, constant(200))
.setHeader(Exchange.CONTENT_TYPE, constant("text/html"));
logger.info("** Route config ok");
}
then in myCsvHandler -> method doHandleCsvDataVB I explicitly throw an exception for testing
a failure :
#Component
public class MyCsvHandler {
#Inject
AFVINCCrudRepository _entityManagerVINC;
#Inject
AFFileCrudRepository _entityManagerAfFile;
#Transactional(propagation = Propagation.REQUIRED , transactionManager="DbTransactionManagerVB")
public void doHandleCsvDataVB(List<List<String>> csvData) throws FileNotEvaluableException
{
//System.out.println("stampo..");
if (csvData.size() > 0){
AfFileEntity afbean = new AfFileEntity();
afbean.setNomeFile("test");
afbean.setDataImport(new java.sql.Timestamp(System.currentTimeMillis()));
afbean.setTipoFile("M");
afbean.setAfStatoFlusso("I");
_entityManagerAfFile.save(afbean);
long pkfile = afbean.getId();
System.out.println("pkfile: " + pkfile);
int i = 1; VincEntity vincBean = new VincEntity();
System.out.println(csvData.size());
for (List<String> rows : csvData){
..
_entityManagerVINC.save(..);
}
throw new FileNotEvaluableException("Il file non è nè una ...");
}
}
}
}
The save methods loop and continue to save data on db..
What's wrong?
Thanks so much.

Answer 1

In your code, you cannot have from inside another from.
from("direct:csvprocessor")
.streamCaching()
.from("file:src/main/resources/data?move=OUT/VB/")
Your second from("file:...") should instead use an enricher. http://camel.apache.org/content-enricher.html
Not sure, but in your doHandleCsvDataVB() method I don't see a try() catch() block. You are going straight from a for loop to throwing the exception.

Answer 2

I did the below way to move the file from one folder to another folder.
#Autowired
FileProcessor fileProcessor;
#Component
public class FileRouteBuilder extends RouteBuilder {
#Override
public void configure() throws Exception {
from("file://{{dir.in}}?readLock=changed&include=.*.csv"&preMove={{dir.progress}}&move={{dir.out}}&moveFailed={{dir.error}}")
.process(fileProcessor).setId("Poller");
}
}
dir.in -Source directory for file polling
dir.progress -File will be moved from in to progress for processing
dir.out -File will be moved here when processed successfully
dir.error - File moved here when any error occur during process
readLock=changed - this will lock the file while reading
include - this will read only csv files in source dir i.e in folder
you may or may not have processor as per your needs.
if you don't want processor then use below code
from("file://{{dir.in}}?readLock=changed&include=.*.csv"&preMove={{dir.progress}}&move={{dir.out}}&moveFailed={{dir.error}}")
.setId("Poller");
Please note that, if you are not using preMove then default .camel folder will be created in the 'in' folder and one copy of file moved to .camel folder. file will always exist in the directory until and unless we remove manually.
So i suggest you to use the preMove in the route.