Camel CsvDataFormat unmarshal multiple issues with Excel csv files - csv

Question

I need to process a csv file obtained from a government site. The file has two different format issues that cannot both be handled by Camel CsvDataFormat unmarshal. Minimal test file:
Registration No,Trade Name
"A009928","Rotagen "Combo""
"A010343","Vet Direct Abamectin Wormer, Bot + Tape"
Using this code to unmarshal:
CsvDataFormat csv = new CsvDataFormat();
csv.setDelimiter(",");
csv.setQuoteDisabled(true);
csv.setUseMaps(false);
from("file://c:/temp?fileName=test.csv&noop=true")
.unmarshal(csv)
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
List<List<String>> rows = (List<List<String>>) exchange.getIn().getBody();
for (int j = 0; j< rows.size();j++) {
List<String> row = rows.get(j);
for (int i = 0; i< row.size();i++) {
log.info("ITEM["+row.get(i)+"]");
}
}
}
});
When setQuoteDisabled(false) I get:
java.lang.IllegalStateException: IOException reading next record: java.io.IOException: (line 2) invalid char between encapsulated token and delimiter
When setQuoteDisabled(true) the file is unmarshaled but the 3rd line ends an additional split at the extra ','
Here's the output:
13:10| INFO | MainRoute.java 54 | ITEM[Registration No]
13:10| INFO | MainRoute.java 54 | ITEM[Trade Name]
13:10| INFO | MainRoute.java 54 | ITEM["A009928"]
13:10| INFO | MainRoute.java 54 | ITEM["Rotagen "Combo""]
13:10| INFO | MainRoute.java 54 | ITEM["A010343"]
13:10| INFO | MainRoute.java 54 | ITEM["Vet Direct Abamectin Wormer]
13:10| INFO | MainRoute.java 54 | ITEM[ Bot + Tape"]
How to configure CsvDataFormat to unmarshall both rows correctly?

Answer 1

Well, this is a problem of CSV as a "soft standard". Rows and delimiters are more or less standardized, but when it comes to quotes, it gets complicated.
Since your data is quoted (i.e. every field value is in quotes), the correct configuration would be
setQuoteDisabled(false)
The second record works fine with this configuration.
"A010343","Vet Direct Abamectin Wormer, Bot + Tape"
Because the fields are enclosed in quotes, the comma inside the data is no problem.
However, the first record contains quotes inside the data.
"A009928","Rotagen "Combo""
According to RFC-4180, Paragraph 2.7 such quotes must be escaped with an additional quote.
If double-quotes are used to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote.
"A009928","Rotagen ""Combo"""
You could try to fix this manually in one record to see if it works like this.
Generally, you have multiple options:
Inform the data provider that his data is not RFC-4180 compliant and ask him to fix it
Fix the data upfront before you read it with Camel
Parse the data by yourself and compensate the quote problem

Answer 2

The second line of your .csv file violates the rules of quoting in csv, or at least as it is understood by the default options of commons-csv (the library camel uses under the hood for these types of things).
The default way of dealing with quotes inside quotes, is to escape the inner quotes by repeating it twice. Keep setQuoteDisabled(false) and correct the second line in your .csv-file to:
"A009928","Rotagen ""Combo"""