Storing Document with generic JsonNode in ElasticSearch using Spring Data - json

Question

I want to store unspecified json payload data in an Elasticsearch index using Spring Data with the following entity
#Document(indexName = "message")
public class Message {
#Id
private String id;
private JsonNode payload;
//getters and setters here
}
The payload varies and needs to be stored in a generic way that can also be easily loaded again that's why I'd like to use the JsonNode here.
A document with "id" gets written but the field "payload" is empty.
When I look up the document written to the index in Kibana it looks like this:
_class:
com.tryout.Message
payload:
id:
30243006-0844-4438-a7f0-db93518b340f
_id:
30243006-0844-4438-a7f0-db93518b340f
_type:
_doc
_index:
message
_score:
0
In the index mapping "payload" also wasn't created and it looks like this:
{
"mappings": {
"_doc": {
"properties": {
"_class": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
Any ideas how I can get my generic payload stored?
(I'm using Elastic v7.9.2 & Spring Booot + spring-boot-data-jpa v2.3.5 and spring-data-elasticsearch v4.1.1)

Answer 1

I had this problem after upgrading spring-data-elastic-search version to 4, and unfortunately I didn't find a clear answer for it. Reading spring data elasticsearch documentation,I found out this:
As of version 4.0 only the Meta Object Mapping is used, the Jackson based mapper is not available anymore and the MappingElasticsearchConverter is used.
After searching several days, finally I solve my problem this way, I hope it could help others.
p.s : JsonNode can include jsonArray or JsonObject, so these two data-types should be handled while reading/writing.
class JsonElasticSearchConverter extends MappingElasticsearchConverter {
private CustomConversions conversions = new ElasticsearchCustomConversions(Collections.emptyList());
CustomElasticSearchConverter(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext) {
super(mappingContext);
setConversions(conversions);
}
CustomElasticSearchConverter(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext, GenericConversionService conversionService) {
super(mappingContext, conversionService);
setConversions(conversions);
}
#Override
protected <R> R readValue(#Nullable Object source, ElasticsearchPersistentProperty property,
TypeInformation<R> targetType) {
if (source == null) {
return null;
}
if (targetType.getType() == JsonNode.class) {
return (R) mapStoredValueToJsonNode(source);
}
return super.readValue(source, property, targetType);
}
#Override
protected Object getWriteComplexValue(ElasticsearchPersistentProperty property, TypeInformation<?> typeHint, Object value) {
if (typeHint.getType() == JsonNode.class) {
return getWriteJsonNodeValue(value);
}
return super.getWriteComplexValue(property, typeHint, value);
}
private JsonNode mapStoredValueToJsonNode(Object source) {
JsonNode jsonNode = null;
ObjectMapper mapper = new ObjectMapper();
if (source instanceof ArrayList) {
ArrayNode array = mapper.valueToTree(source);
jsonNode = mapper.valueToTree(array);
}
if (source instanceof HashMap) {
jsonNode = mapper.convertValue(source, JsonNode.class);
}
return jsonNode;
}
private Object getWriteJsonNodeValue(Object value) {
JsonNode jsonNode = null;
ObjectMapper mapper = new ObjectMapper();
if (value instanceof ObjectNode)
try {
jsonNode = mapper.readTree(value.toString());
} catch (IOException shouldNotHappened) {
//log.warn("This error should not happened" + shouldNotHappened.getMessage());
}
else if (value instanceof ArrayNode) {
ArrayNode array = mapper.valueToTree(value);
jsonNode = mapper.valueToTree(array);
}
return jsonNode;
}
}