Trying to parse .csv file with jackson-dataformat-csv. File contains a lot of columns not relevant for my program.
Tried to use #JsonIgnoreProperties(ignoreUnknown = true) on my data class,
and csvMapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES), but neither works, and application throws exception:
com.fasterxml.jackson.databind.RuntimeJsonMappingException: Too many entries: expected at most 2 (value #2 (17 chars) "policy_issue_date")
at [Source: (com.fasterxml.jackson.dataformat.csv.impl.UTF8Reader); line: 1, column: 37]
at com.fasterxml.jackson.databind.MappingIterator.next(MappingIterator.java:194)
at pl.polins.readers.oc.OcPolicyCsvReader.readNext(OcPolicyCsvReader.kt:25)
at pl.polins.readers.oc.OcPolicyCsvReaderTest.should read PolicyCsv from .csv file(OcPolicyCsvReaderTest.groovy:19)
Caused by: com.fasterxml.jackson.dataformat.csv.CsvMappingException: Too many entries: expected at most 2 (value #2 (17 chars) "policy_issue_date")
at [Source: (com.fasterxml.jackson.dataformat.csv.impl.UTF8Reader); line: 1, column: 37]
at com.fasterxml.jackson.dataformat.csv.CsvMappingException.from(CsvMappingException.java:23)
at com.fasterxml.jackson.dataformat.csv.CsvParser._reportCsvMappingError(CsvParser.java:1210)
at com.fasterxml.jackson.dataformat.csv.CsvParser._handleExtraColumn(CsvParser.java:965)
at com.fasterxml.jackson.dataformat.csv.CsvParser._handleNextEntry(CsvParser.java:826)
at com.fasterxml.jackson.dataformat.csv.CsvParser.nextToken(CsvParser.java:580)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:418)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1266)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:325)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
at com.fasterxml.jackson.databind.MappingIterator.nextValue(MappingIterator.java:277)
at com.fasterxml.jackson.databind.MappingIterator.next(MappingIterator.java:192)
... 2 more
Is there any solution to ignore unwanted columns in csv?
Found solution:
csvMapper.enable(CsvParser.Feature.IGNORE_TRAILING_UNMAPPABLE)
This worked for me:
csvMapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
Introduction
For the sake of comprehensibility, here is a simple (Java) example that:
read a CSV-file from an InputStream (jackson-dataformat-csv
dependency)
map its content to a list of objects (jackson-core dependency)
CSV file content
Let be data.csv a CSV file with the following data:
a;b;c
1;2;0.5
3;4;
Java Class Data with a missing attributes
The class MyModel represents a data class with the following attributes:
private Long a;
private Integer b;
Note the attribute c is missing, hence the parser will have to ignore it.
Read the CSV content and map into a list of objects
So the CsvMapper can be coupled with the Jackson object mapper to read records from a CSV file to a list of objects, the whole in a convenient method:
<U> List<U> mapRecordsToObjects(InputStream inputStream, Class<U> encodingType) {
CsvMapper csvMapper = new CsvMapper();
CsvSchema bootstrapSchema = CsvSchema.emptySchema() //
.withHeader() //
.withColumnSeparator(";");
ObjectReader reader = csvMapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES) //
.readerFor(encodingType) //
.with(bootstrapSchema);
MappingIterator<U> iterator;
try {
iterator = reader.readValues(inputStream);
} catch (IOException e) {
throw new IllegalStateException(String.format("could not access file [%s]", this.source), e);
}
List<U> results = new ArrayList<>();
iterator.forEachRemaining(results::add);
return results;
}
Finally, let's call this method:
List<MyModel> result = mapRecordsToObjects(fileInputStream, MyModel.class);
In order to read the file, you will just need to initialise the InputStream.
Deserialization feature
In the documentation, the class DeserializationFeature has the following description:
Enumeration that defines simple on/off features that affect the way
Java objects are deserialized from JSON
In this enumeration class, there are many feature with a default state (sometimes per default enabled, sometimes disabled). The feature FAIL_ON_UNKOWN_PROPERTIES is disable per default can can be enabled as shown in the example. In its description, one can read:
Feature that determines whether encountering of unknown properties
(ones that do not map to a property, and there is no "any setter" or
handler that can handle it) should result in a failure (by throwing a
{#link JsonMappingException}) or not.
Related
I'm using xStream to some JSON. I've used xstream quite extensively over the years. However this issue has me stumped.
I'm getting the following ConversionException...
com.thoughtworks.xstream.converters.ConversionException: For input string: ".232017E.232017E44"
---- Debugging information ----
message : For input string: ".232017E.232017E44"
cause-exception : java.lang.NumberFormatException
cause-message : For input string: ".232017E.232017E44"
class : java.sql.Timestamp
required-type : java.sql.Timestamp
converter-type : com.etepstudios.xstream.XStreamTimestampConverter
line number : -1
class[1] : com.pbp.bookacall.dataobjects.AppleReceipt
converter-type[1] : com.thoughtworks.xstream.converters.reflection.ReflectionConverter
class[2] : com.pbp.bookacall.dataobjects.AppleReceiptCollection
version : 1.4.10
-------------------------------
at com.etepstudios.xstream.XStreamTimestampConverter.unmarshal(XStreamTimestampConverter.java:87)
In my XStreamTimestampConverter class I print out the value that is attempting to be converted.. Which turns out to be the following...
XStreamTimestampConverter value = 2017-08-05 23:44:23.GMT
Here is the unmarshal function in my converter...
public Object unmarshal(HierarchicalStreamReader reader, UnmarshallingContext context)
{
Timestamp theTimestamp;
Date theDate;
String value = reader.getValue ();
try
{
SimpleDateFormat formatter = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.Z");
theDate = formatter.parse(value);
theTimestamp = new Timestamp (theDate.getTime());
}
catch (Exception e)
{
System.out.println ("XStreamTimestampConverter value = " + value);
throw new ConversionException(e.getMessage(), e);
}
return theTimestamp;
}
Any idea where this odd string is coming from? It does not exist anywhere in my JSON. Does xstream have some odd .[num]E.[num]E[num] notation for something? These numbers can change as I run this each time. Also I get an For input string: "" on occasion too. Yet the value is similar to the what is above. It's like it's randomly getting odd values for somewhere.
The data source is from Apple's In-App Purchase /VerifyReceipt web call. The system works just fine some times but then others it does not. It's also important to note that in this very case it parsed 100s of other Date/Timestamp strings using this converter. It just get's confused. Perhaps due to the size of the data?
So I figured out what was going on here. The unmarshal function above is not exactly as I have it in code...
The SimpleDateFormat formatter is actually set in the class rather than in the unmarshal method. Therefore if Xstream holds on to an instance of my converter and the unmarshal is called across multiple threads then it is possible that the formatter can get confused since it is the same object.
That's my only guess at this point as moving the formatter initialization into the method solved the issue. I would say SimpleDateFormatter is not thread safe?
It was the just the sheer about of data and the number of times it was concurrently being called that exposed this issue. Just a tip for anyone else in case this happens to them.
Similar to Jersey: Json array with 1 element is serialized as object BUT on the client side. E.g. I recieve a JSON object where a field is an array regulary, but in case there is only one element, it is a single object.
{"fileInfo":[{"fileName":"weather.arff","id":"10"},"fileName":"supermarket.arff","id":"11"}]}
versus
{"fileInfo":{"fileName":"weather.arff","id":"10"}}
I'm parsing/unmarshalling the JSON using Jersey/Genson. Of course, if the JSON doesnt match the target class I recieve an error (such as expected [ but read '{' )
I've read a lot about this bug and how to avoid when creating JSON objects on the SERVER side, but I found nothing about how to handle this issus when dealing on the CLIENT side.
As always, I prefere the most codeless possibility if there are several solutions...
BTW: Moxy works but it does not marshal native Object-type objects which is another requirement...
Update
Starting with Genson 1.3 release you can achieve it by enabling permissiveParsing:
Genson genson = new GensonBuilder().usePermissiveParsing(true).create();
Answer
Uh, do you know what library produces this on server side? I am curious to see who is responsible for all those badly structured jsons out there...
It is not yet supported in Genson. Originally because IMO people should not produce such dynamic json. Anyway, I opened an issue - this can be easily done, you can expect it to be present in the release coming next week.
Otherwise here is a way to achieve it without breaking the existing mechanisms.
You need to register a Factory that will use Gensons collections factory to create an instance of its standard collection converter. Then you will wrap this converter in another one that will handle the object to array logic. Here is the code (not codeless..., but if you wait a bit you won't have to code :)).
import com.owlike.genson.convert.DefaultConverters.CollectionConverterFactory;
import com.owlike.genson.convert.DefaultConverters.CollectionConverterFactory;
class SingleObjectAsCollectionFactory implements Factory<Converter<Collection>> {
// get the default factory
Factory<Converter<Collection<?>>> defaultFactory = CollectionConverterFactory.instance;
#Override
public Converter<Collection> create(Type type, Genson genson) {
// obtain an instance of the correct default converter for this type
final CollectionConverter defaultConverter = (CollectionConverter) defaultFactory.create(type, genson);
// wrap it in your own converter
return new Converter<Collection>() {
#Override
public void serialize(Collection object, ObjectWriter writer, Context ctx) throws Exception {
defaultConverter.serialize(object, writer, ctx);
}
#Override
public Collection deserialize(ObjectReader reader, Context ctx) throws Exception {
if (reader.getValueType() == ValueType.OBJECT) {
Object object = defaultConverter.getElementConverter().deserialize(reader, ctx);
Collection result = defaultConverter.create();
result.add(object);
return result;
} else return defaultConverter.deserialize( reader, ctx );
}
};
}
}
And then register it
Genson genson = new GensonBuilder()
.withConverterFactory(new SingleObjectAsCollectionFactory())
.create();
I'm stumped as to what may be the problem. I'm using GSON (2.2.4) to serialize/deserialize simple json objects and arrays, which I display in a JTable. Everything works fine but when I feed a json object like this:
{"1":{"336":"#1700EB","17":"#EB0000","10":"#EB0000","26":"#1700EB","3":"#1700EB","1":"#EB0000"}}
it throws this error:
com.google.gson.JsonSyntaxException: java.io.EOFException: End of input at line 1 column 71
at com.google.gson.Gson.fromJson(Gson.java:813)
at com.google.gson.Gson.fromJson(Gson.java:768)
at com.google.gson.Gson.fromJson(Gson.java:717)
When I remove the pound signs, it functions normally.
Here's the code that does the conversion:
Map<String, String> conv = null;
Type type = new TypeToken<Map<String, String>>(){}.getType();
ListMultimap<String, Object> returnMap = ArrayListMultimap.create();
try {
conv = g.fromJson(parse, type);
for(String key : conv.keySet()) {
returnMap.put("Key", key);
returnMap.put("Value", conv.get(key));
}
} catch (JsonSyntaxException jse) {
type = new TypeToken<Map<String, Object>>(){}.getType();
conv = g.fromJson(parse, type);
for(Object key : conv.keySet()) {
returnMap.put("Key", key);
returnMap.put("Value", conv.get(key));
}
}
Please note that I am working on a "legacy" application and I have little control over the values that come to the part of the code I'm working on; which is why I have that odd try-catch block.
Most users of this application are not savvy enough to treat their strings/jsons with the express purpose of avoiding tripping exceptions like the one outlined here (e.g. not including the # sign when passing it through the application; but adding it back when they need it), so I'd really like to fix it within the code.
Thanks!
Edit: I forgot to add an important detail. What I'm doing with the code is display data in tabular form. When a user selects a cell, it handles according to context. In the context of a cell containing a json object or array, it uses the code above to extract the values from the json and passes it as the new table data for the table's model. So, the sample json object should ideally come out like this (imagine table form)
336 | #1700EB
17 | #EB0000
10 | #EB0000
26 | #1700EB
3 | #1700EB
1 | #EB0000
...but it doesn't get there. The previous table that had the cell with the json object looked like this:
1 | {336=#1700EB, 17=#EB0000, 10=#EB0000, 26=#1700EB, 1=#EB0000}
Does this form have anything to do with the error? I understand that the json object form should be like this (at least the ones I work with): {"336":"#1700EB"...}. That's probably my only hunch as to what may be wrong.
Thanks for trying guys. I was able to fix the problem though, honestly, I still don't know the underlying cause. Anyway, I refactored the code to not do the initial attempt to map using Type. Apparently, it caused the problem further downstream.
Here's the amended code for the curious:
Map<String, Object> conv = null;
Type type = new TypeToken<Map<String, Object>>(){}.getType();
ListMultimap<String, Object> returnMap = ArrayListMultimap.create();
conv = g.fromJson(parse, type);
for(Object key : conv.keySet()) {
returnMap.put("Key", key);
Object check = conv.get(key.toString());
if ((check.toString().startsWith("{") && check.toString().endsWith("}")) ||
(check.toString().startsWith("[") && check.toString().endsWith("]")))
check = g.toJson(check);
returnMap.put("Value", check);
}
return returnMap;
I have three levels deep of a hierarchy that I am binding in a JSON request:
Group -> Zone -> Segment
(1) -> (n) -> (n)
In my command object I have:
class GroupCommand {
Long id
Set zones
}
When binding the JSON request the zones get bound properly and I get a LinkedHashSet that I can get the properties of and use with my domain object. However when I get to iterating over the segments in my service:
groupCommand.zones.each { zone ->
zone.segments.each { segment ->
//Would like to get LinkedHashMap here also
//but get JSONArray
}
}
As noted above, I'd ideally like the deeply nested Segments to also bind to a LinkedHashMap but it's bound to a JSONArray.
Any suggestions how to get it bound to a LinkedHashMap as I'd like to avoid having to manipulate JSONArray in my service and thereby coupling my service with the JSON format.
If there's a way to do the conversion at the command level using a getter I'm all for that also.
thanks
EDIT:
Using
List zones = org.apache.commons.collections.list.LazyList.decorate(new ArrayList(), new org.apache.commons.collections.functors.InstantiateFactory(ZoneCommand.class))
appears to work but the underlying objects are still JSON elements. I then tried using:
List<RateZoneCommand> zones = org.apache.commons.collections.list.LazyList.decorate(new ArrayList(), new org.apache.commons.collections.functors.InstantiateFactory(ZoneCommand.class))
and at least I got an error indicating it trying to convert:
Validation error: ... org.codehaus.groovy.grails.web.json.JSONArray to required type java.util.List for property zones; nested exception is java.lang.IllegalStateException: Cannot convert value of type [org..JSONObject] to required type [ZoneCommand] for property zones[0]: no matching editors or conversion strategy found.
Create a command class for each level. Mark Zone- and Segment-command as #Validateable.
To your GroupCommand, add:
List zones = org.apache.commons.collections.list.LazyList.decorate(new ArrayList(), new org.apache.commons.collections.functors.InstantiateFactory(ZoneCommand.class))
To your ZoneCommand, add:
List segments = org.apache.commons.collections.list.LazyList.decorate(new ArrayList(), new org.apache.commons.collections.functors.InstantiateFactory(SegmentCommand.class))
In your form just use group.zones[0].segments[0]. If you change a field type of your command class, remember to restart the grails server.
I'm aware that there are multiple way to tell JacksonJson to ignore
properties during rendering but all of them are static. (JasonIgnore, MixIn classes, ..).
This is my scenario. A domain object can implement a interface called FilteredDomain to
allow it to be dynamically filtered. The interface is simple and only exposes one method
"getIgnoreProperties". (A list of properties to ignore).
I then register a Custom Serializer that binds to the FilteredDomain object. The
code looks something like:
private class FilteredDomainSerializer extends JsonSerializer<FilteredDomain> {
public void serialize(FilteredDomain arg, JsonGenerator jgen,
SerializerProvider provder) throws IOException,
JsonProcessingException {
final BeanWrapper wrapper = PropertyAccessorFactory.forBeanPropertyAccess(arg);
for (PropertyDescriptor pd : wrapper.getPropertyDescriptors()) {
final String name = pd.getName();
if (arg.getIgnoreProperties().containsKey(name))
continue;
final Object value = wrapper.getPropertyValue(name);
jgen.writeObjectField(name, value);
}
}
}
First, I really dislike that I need to use the Spring Bean wrapper to get a list of all properties and iterate through them (There must be a way to do this is jackson json).
Second, The code still dosen't work. I get the error:
org.codehaus.jackson.JsonGenerationException: Can not write a field name, expecting a value
at org.codehaus.jackson.impl.JsonGeneratorBase._reportError(JsonGeneratorBase.java:480)
at org.codehaus.jackson.impl.Utf8Generator.writeFieldName(Utf8Generator.java:270)
at org.codehaus.jackson.JsonGenerator.writeObjectField(JsonGenerator.java:1088)
at com.rootmusic.util.SystemJsonObjectMapper$ValueObjectSerializer.serialize(SystemJsonObjectMapper.java:65)
at com.rootmusic.util.SystemJsonObjectMapper$ValueObjectSerializer.serialize(SystemJsonObjectMapper.java:1)
at org.codehaus.jackson.map.ser.ContainerSerializers$IndexedListSerializer.serializeContents(ContainerSerializers.java:304)
at org.codehaus.jackson.map.ser.ContainerSerializers$IndexedListSerializer.serializeContents(ContainerSerializers.java:254)
at org.codehaus.jackson.map.ser.ContainerSerializers$AsArraySerializer.serialize(ContainerSerializers.java:142)
at org.codehaus.jackson.map.ser.MapSerializer.serializeFields(MapSerializer.java:287)
at org.codehaus.jackson.map.ser.MapSerializer.serialize(MapSerializer.java:212)
at org.codehaus.jackson.map.ser.MapSerializer.serialize(MapSerializer.java:23)
at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:606)
at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:280)
The error comes from the fact that you are not writing START_OBJECT / END_OBJECT around field-name/value pairs, so that should be easy to fix.
As to more dynamic filtering, you could read this blog entry which includes standard methods. #JsonView works if you have sets of static definitions (one of which you can dynamically select on per-serialization basis), but if you want yet more dynamic system, #JsonFilter is the way to go.
Alternatively, another relatively simple way would be to first "convert" your POJO into a Map:
Map props = objectMapper.convertValue(pojo, Map.class);
(which is similar to serializing it as JSON, except that result is a Map which would render as JSON)
and then selectively trim Map, and serialize that as JSON. Or, if you prefer, you can use JsonNode ("tree model") as the intermediate thing to modify and then serialize.