Jackson Kotlin - Deserialize JsonNode - json

Problem
I have JSON content in the form of a string, which I first want to traverse programmatically with Jackson. Then, when I have the node of interest, I want to deserialize it.
What I have tried
I have successfully deserialized strings using mapper.readValue, but now I want to perform such an operation on a jsonNode instead of a string.
Libraries
jackson-core:2.9.9
jackson-module-kotlin:2.9.9
Kotlin 1.3.41
kotlin-stdlib-jdk8:1.3.41
Code
package somepackage
import com.fasterxml.jackson.databind.JsonNode
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.kotlin.KotlinModule
import com.fasterxml.jackson.module.kotlin.readValue
import com.fasterxml.jackson.module.kotlin.treeToValue
fun main() {
val mapper = ObjectMapper().registerModule(KotlinModule())
readValueWorksFine(mapper)
treeToValueFails(mapper)
}
fun treeToValueFails(mapper: ObjectMapper) {
val fullJsonContent = """
[{
"product_id":123,
"Comments":
[{
"comment_id": 23,
"message": "Hello World!"
}]
}]
""".trimIndent()
// Traverse to get the node of interest
val commentsNode: JsonNode = mapper.readTree(fullJsonContent).get(0).get("Comments")
// Deserialize
val comments: List<Comment> = mapper.treeToValue<List<Comment>>(commentsNode)
// The line below fails. (I would have expected the exception to be thrown in the line above instead.
// Exception:
// Exception in thread "main" java.lang.ClassCastException: class
// java.util.LinkedHashMap cannot be cast to class somepackage.Comment (java.util.LinkedHashMap is in module
// java.base of loader 'bootstrap'; somepackage.Comment is in unnamed module of loader 'app')
for (comment: Comment in comments) { // This line fails
println(comment.comment_id)
println(comment.message)
}
}
fun readValueWorksFine(mapper: ObjectMapper) {
val commentsJsonContent = """
[{
"comment_id": 23,
"message": "Hello World!"
}]
""".trimIndent()
val comments1: List<Comment> = mapper.readValue<List<Comment>>(commentsJsonContent)
for (comment in comments1) {
println(comment)
}
}
data class Comment(val comment_id: Long, val message: String)
Exception/Output
The code above results in the following exception/output:
Comment(comment_id=23, message=Hello World!)
Exception in thread "main" java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class somepackage.Comment (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; somepackage.Comment is in unnamed module of loader 'app')
at somepackage.TKt.treeToValueFails(T.kt:39)
at somepackage.TKt.main(T.kt:13)
at somepackage.TKt.main(T.kt)

The problem cause
Even though ObjectMapper.treeToValue is a Kotlin inline extension function with a reified generic parameter (which means that generics are preserved at runtime), it calls the Java ObjectMapper.treeToValue(TreeNode, Class<T>) method. The value passed as Class<T> will loose generic type information for generic types such as List<Comment>, because of type erasure.
So treeToValue can be used for:
mapper.treeToValue<Comment>(commentNode)
but not for:
mapper.treeToValue<List<Comment>>(commentsNode)
Also note that ObjectMapper contains multiple methods that have #SuppressWarnings annotations, which causes some problems not to appear at compile-time, but at run-time.
Solution 1 - use convertValue()
This is the best solution. It uses the Kotlin extension function ObjectMapper.convertValue.
val commentsNode = mapper.readTree(fullJsonContent).get(0).get("Comments")
val comments = mapper.convertValue<List<Comment>>(commentsNode)
Solution 2 - use an ObjectReader
This solution doesn't use jackson-module-kotlin extension functions.
val reader = mapper.readerFor(object : TypeReference<List<Comment>>() {})
val comments: List<Comment> = reader.readValue(commentsNode)
Solution 3 - deserialize in map
Because treeToValue (Kotlin extension function) does work for non-generic types, you can first get the nodes as as list of JsonNodes, and then map each JsonNode to a Comment.
But it's cumbersome that you cannot simply return mapper.treeToValue(it), because that causes type inference compile errors.
val commentsNode = mapper.readTree(fullJsonContent).get(0).get("Comments")
val comments = commentsNode.elements().asSequence().toList().map {
val comment: Comment = mapper.treeToValue(it)
comment
}

Related

Deserializing JSON into Serializable class with generic field - error: Star projections in type arguments are not allowed

Intro
I'm sending JSON messages between two backend servers that use different languages. The producing
server creates a variety of JSON messages, wrapped inside a message with metadata.
The wrapping class is Message, The consuming server has to determine which type of message its
receiving based solely on the message contents.
When I try to use a star-projection to
deserialize the message, I get an error.
Example
import kotlinx.serialization.json.Json
#Language("JSON")
val carJson = """
{
"message_type": "some message",
"data": {
"info_type": "Car",
"name": "Toyota"
}
}
""".trimIndent()
// normally I wouldn't know what the Json message would be - so the type is Message<*>
val actualCarMessage = Json.decodeFromString<Message<*>>(carJson)
Error message
Exception in thread "main" java.lang.IllegalArgumentException: Star projections in type arguments are not allowed, but Message<*>
at kotlinx.serialization.SerializersKt__SerializersKt.serializerByKTypeImpl$SerializersKt__SerializersKt(Serializers.kt:81)
at kotlinx.serialization.SerializersKt__SerializersKt.serializer(Serializers.kt:59)
at kotlinx.serialization.SerializersKt.serializer(Unknown Source)
at ExampleKt.main(example.kt:96)
at ExampleKt.main(example.kt)
Class structure
I want to deserialize JSON into a data class, Message, that has a field with a generic type.
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
#Serializable
data class Message<out DataType : SpecificInformation>(
#SerialName("message_type")
val type: String,
#SerialName("data")
val data: DataType,
)
The field is constrained by a sealed interface, SpecificInformation, with some implementations.
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.JsonClassDiscriminator
#JsonClassDiscriminator("info_type")
sealed interface SpecificInformation {
#SerialName("info_type")
val infoType: String
}
#Serializable
#SerialName("User")
data class UserInformation(
#SerialName("info_type")
override val infoType: String,
val name: String,
) : SpecificInformation
// there are more implementations...
Workaround?
This is a known
issue (kotlinx.serialization/issues/944)
,
so I'm looking for workarounds.
I have control over the JSON structure and libraries - though I have a preference for
kotlinx.serialization.
I can't change that there are two JSON objects, one is inside the other, and the discriminator is
inside the inner-class.
A custom serializer would be great. But I'd prefer to have this configured on the class or file
(with #Serializable(with = ...) or #file:UseSerializers(...)) as using a
custom SerializersModule is not as seamless.
Attempt: JsonContentPolymorphicSerializer
I've written a custom serializer, which only if it's used specifically (which is something I'd like
to avoid). It's also quite clunky, breaks if the data classes change or a new one is added, and
doesn't benefit from the sealed interface.
Can this be improved so that
It can be used generically? Json.decodeFromString<Message<*>>(carJson)
It doesn't have any hard-coded strings?
class MessageCustomSerializer : JsonContentPolymorphicSerializer<Message<*>>(Message::class) {
override fun selectDeserializer(element: JsonElement): DeserializationStrategy<out Message<*>> {
val discriminator = element
.jsonObject["data"]
?.jsonObject?.get("info_type")
?.jsonPrimitive?.contentOrNull
println("found discriminator $discriminator")
val subclassSerializer = when (discriminator?.lowercase()) {
"user" -> UserInformation.serializer()
"car" -> CarInformation.serializer()
else -> throw IllegalStateException("could not find serializer for $discriminator")
}
println("found subclassSerializer $subclassSerializer")
return Message.serializer(subclassSerializer)
}
}
fun main() {
#Language("JSON")
val carJson = """
{
"message_type": "another message",
"data": {
"info_type": "Car",
"brand": "Toyota"
}
}
""".trimIndent()
val actualCarMessage =
Json.decodeFromString(MessageCustomSerializer(), carJson)
val expectedCarMessage = Message("another message", CarInformation("Car", "Toyota"))
require(actualCarMessage == expectedCarMessage) {
println("car json parsing ❌")
}
println("car json parsing ✅")
}
#Serializable(with = ... - infinite loop
I tried applying MessageCustomSerializer directly to Message...
#Serializable(with = MessageCustomSerializer::class)
data class Message<out T : SpecificInformation>(
//...
But then I couldn't access the plugin-generated serializer, and this causes an infinite loop.
return Message.serializer(subclassSerializer) // calls 'MessageCustomSerializer', causes infinite loop
#Serializer(forClass = ...) - not generic
In addition to annotating Message with #Serializable(with = MessageCustomSerializer::class), I
tried
deriving a plugin-generated serializer:
#Serializer(forClass = Message::class)
object MessagePluginGeneratedSerializer : KSerializer<Message<*>>
But this serializer is not generic, and causes an error
java.lang.AssertionError: No such value argument slot in IrConstructorCallImpl: 0 (total=0).
Symbol: MessageCustomSerializer.<init>|-5645683436151566731[0]
at org.jetbrains.kotlin.ir.expressions.IrMemberAccessExpressionKt.throwNoSuchArgumentSlotException(IrMemberAccessExpression.kt:66)
at org.jetbrains.kotlin.ir.expressions.IrFunctionAccessExpression.putValueArgument(IrFunctionAccessExpression.kt:31)
at org.jetbrains.kotlinx.serialization.compiler.backend.ir.IrBuilderExtension$DefaultImpls.irInvoke(GeneratorHelpers.kt:210)
at org.jetbrains.kotlinx.serialization.compiler.backend.ir.SerializableCompanionIrGenerator.irInvoke(SerializableCompanionIrGenerator.kt:35)
You are asking many things here, so I will simply try to give some pointers in regards to the errors you are making which you seem to be stuck on. With those in mind, and reading the documentation I link to, I believe you should be able to resolve the rest yourself.
Polymorphic serialization
Acquaint yourself with kotlinx.serialization polymorphic serialization. When you are trying to serialize Message<*> and DataType you are trying to use polymorphic serialization.
In case you are serializing Message<*> as the root object, specifying PolymorphicSerializer explicitly (as I also posted in the bug report you link to) should work. E.g., Json.decodeFromString( PolymorphicSerializer( Message::class ), carJson ).
P.s. I'm not 100% certain what you are trying to do here is the same as in the bug report. Either way, specifying the serializer explicitely should work, whether or not it is a bug that you shouldn't be required to do so.
The message_type and info_type fields you have in Message and DataType respectively are class discriminators. You need to configure this in your Json settings, and set the correct SerialName on your concrete classes for them to work. Using a different class discriminator per hierarchy is only possible starting from kotlinx.serialization 1.3.0 using #JsonClassDiscriminator.
Overriding plugin-generated serializer
But then I couldn't access the plugin-generated serializer, and this causes an infinite loop.
#Serializable(with = ...) overrides the plugin-generated serializer. If you want to retain the plugin-generated serializer, do not apply with.
When you are serializing the object directly (as the root object), you can still pass a different serializer to use as the first parameter to encode/decode. When you want to override the serializer to use for a specific property nested somewhere in the root object, use #Serializable on the property.
Polymorphism and generic classes
The "No such value argument slot in IrConstructorCallImpl: 0" error is to be expected.
You need to do more work in case you want to specify a serializer for polymorphic generic classes.

jackson databind Json serialization is writing Some(99): Option[Int] as {"empty":true,"defined":false}

I'm using Jackson in my Scala/Spark program and I've distilled my issue to the simple example
below. My problem is that when my case class has the Option[Int] field (age) set to None
I see reasonable deserialization output (that is: a struct with empty=true). However, when
age is defined, i.e., set to some Int like Some(99), I never see the integer value in the
deserialization output .
Given :
import com.fasterxml.jackson.databind.ObjectMapper
import java.io.ByteArrayOutputStream
import scala.beans.BeanProperty
case class Dog(#BeanProperty name: String, #BeanProperty age: Option[Integer])
object OtherTest extends App {
jsonOut(Dog("rex", None))
jsonOut(Dog("mex", Some(99)))
private def jsonOut(dog: Dog) = {
val mapper = new ObjectMapper();
val stream = new ByteArrayOutputStream();
mapper.writeValue(stream, dog);
System.out.println("result:" + stream.toString());
}
}
My output is as shown below. Any hints/help greatly appreciated !
result:{"name":"rex","age":{"empty":true,"defined":false}}
result:{"name":"mex","age":{"empty":false,"defined":true}}
Update after the Helpful Answer
Here are the dependencies that worked for me:
implementation 'org.scala-lang:scala-library:2.12.2'
implementation "org.apache.spark:spark-sql_2.12:3.1.2"
implementation "org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2"
implementation "org.apache.spark:spark-avro_2.12:3.1.2"
implementation 'com.fasterxml.jackson.module:jackson-module-scala_2.12:2.10.0'
Here is the updated code (with frequent flyer bonus - round trip example):
private def jsonOut(dog: Dog) = {
val mapper = new ObjectMapper()
mapper.registerModule(DefaultScalaModule)
val stream = new ByteArrayOutputStream();
mapper.writeValue(stream, dog);
val serialized = stream.toString()
System.out.println("result:" + serialized);
// verify we can read the serialized thing back to case class:
val recovered = mapper.readValue(serialized, classOf[Dog])
System.out.println("here is what we read back:" + recovered);
}
Here is the resultant output (as expected now ;^) ->
> Task :OtherTest.main()
result:{"name":"rex","age":null}
here is what we read back:Dog(rex,None)
result:{"name":"mex","age":99}
here is what we read back:Dog(mex,Some(99))
You need to add the Jackson module for Scala to make it work with standard Scala data types.
Add this module as your dependency: https://github.com/FasterXML/jackson-module-scala
Follow the readme on how to initialize your ObjectMapper with this module.

Serializing Java Path with kotlinx.serialization

With kotlinx.serialization, this code will throw an error:
println(Json.encodeToString(Path.of("value")))
saying kotlinx.serialization.SerializationException: Class 'WindowsPath' is not registered for polymorphic serialization in the scope of 'Path'.
WindowsPath is internal, therefore I can not register it as a polymorphic subclass (as in this example), only with Path itself, and even a custom KSerializer for Path throws the same exact error.
Is there any way to have Path properly serialize/deserialize without having to store it as a string?
Path is an interface, so it's implicitly serializable with the PolymorphicSerializer strategy. This strategy requires you to register serializers for subclasses implementing it, but as you know, in this case it's impossible.
There is a default polymorphic serializer, but it affects only deserialization process and it works only when deserializable value is a JSONObject.
And for the following serializer
object PathAsStringSerializer : KSerializer<Path> {
override val descriptor = PrimitiveSerialDescriptor("Path", PrimitiveKind.STRING)
override fun serialize(encoder: Encoder, value: Path) = encoder.encodeString(value.toAbsolutePath().toString())
override fun deserialize(decoder: Decoder): Path = Path.of(decoder.decodeString())
}
\\Not working
val module = SerializersModule { polymorphicDefault(Path::class) { PathAsStringSerializer } }
val decoded : Path = Json { serializersModule = module }.decodeFromString("C:\\Temp")
it will throw runtime exception kotlinx.serialization.json.internal.JsonDecodingException: Expected class kotlinx.serialization.json.JsonObject as the serialized body of kotlinx.serialization.Polymorphic<Path>, but had class kotlinx.serialization.json.JsonLiteral
So, it can't be serialized in a common way, and there are 3 cases of its serialization/deserialization, which need to be handled:
1. Serialization of simple Path variable
In this case you need to explicitly pass your custom serializer:
val path = Path.of("C:\\Temp")
val message1 = Json.encodeToString(PathAsStringSerializer, path).also { println(it) }
println(Json.decodeFromString(PathAsStringSerializer, message1))
2. Serialization of classes, which use Path as a generic parameter
In this case you need to define separate serializers (you may reference original PathAsStringSerializer) and also explicitly pass them:
object ListOfPathsAsStringSerializer : KSerializer<List<Path>> by ListSerializer(PathAsStringSerializer)
val message2 = Json.encodeToString(ListOfPathsAsStringSerializer, listOf(path)).also { println(it) }
println(Json.decodeFromString(ListOfPathsAsStringSerializer, message2))
#Serializable
data class Box<T>(val item: T)
object BoxOfPathSerializer : KSerializer<Box<Path>> by Box.serializer(PathAsStringSerializer)
val message3 = Json.encodeToString(BoxOfPathSerializer, Box(path)).also { println(it) }
println(Json.decodeFromString(BoxOfPathSerializer, message3))
3. Serialization of classes, which have fields of aforementioned types
In this case you need to add respectful #Serializable(with = ...) annotations for these fields:
#Serializable
data class InnerObject(
#Serializable(with = ListOfPathsAsStringSerializer::class)
val list: MutableList<Path> = mutableListOf(),
#Serializable(with = PathAsStringSerializer::class)
val path: Path,
#Serializable(with = BoxOfPathSerializer::class)
val box: Box<Path>
)
Or just list them once for a whole file:
#file: UseSerializers(PathAsStringSerializer::class, ListOfPathsAsStringSerializer::class, BoxOfPathSerializer::class)
Plugin-generated serializer for this case would be good enough:
val message4 = Json.encodeToString(InnerObject(mutableListOf(path), path, Box(path))).also { println(it) }
println(Json.decodeFromString<InnerObject>(message4))

How do I deserialize JSON into a List<SomeType> with Kotlin + Jackson [duplicate]

This question already has answers here:
How to use jackson to deserialize to Kotlin collections
(3 answers)
Closed 7 years ago.
What is the correct syntax to deserialize the following JSON:
[ {
"id" : "1",
"name" : "Blues"
}, {
"id" : "0",
"name" : "Rock"
} ]
I tried:
//Works OK
val dtos = mapper.readValue(json, List::class.java)
However I want:
val dtos : List<GenreDTO> = mapper.readValue(json,
List<GenreDTO>::class.java)
The above syntax is not correct and gives: only classes are allowed on the left hand side of a class literal
NOTE: The answer from #IRus is also correct, it was being modified at the same time I wrote this to fill in more details.
You should use the Jackson + Kotlin module or you will have other problems deserializing into Kotlin objects when you do no have a default constructor.
Your first sample of the code:
val dtos = mapper.readValue(json, List::class.java)
Is returning an inferred type of List<*> since you did not specify more type information, and it is actually a List<Map<String,Any>> which is not really "working OK" but is not producing any errors. It is unsafe, not typed.
The second code should be:
import com.fasterxml.jackson.module.kotlin.jacksonObjectMapper
import com.fasterxml.jackson.module.kotlin.readValue
val mapper = jacksonObjectMapper()
// ...
val genres: List<GenreDTO> = mapper.readValue(json)
You do not need anything else on the right side of the assignment, the Kotlin module for Jackson will reify the generics and create the TypeReference for Jackson internally. Notice the readValue import, you need that or .* for the com.fasterxml.jackson.module.kotlin package to have the extension functions that do all of the magic.
A slightly different alternative that also works:
val genres = mapper.readValue<List<GenreDTO>>(json)
There is no reason to NOT use the extension functions and the add-on module for Jackson. It is small and solves other issues that would require you to jump through hoops to make a default constructor, or use a bunch of annotations. With the module, your class can be normal Kotlin (optional to be data class):
class GenreDTO(val id: Int, val name: String)
The error you're getting is about following expression:
List<GenreDTO>::class.java
Because of how jvm treats generics there's no separate class for List<GenreDTO> thus compiler complains. Similarly in Java the following will not compile:
List<GenreDTO>.getClass()
Here's a sample that will deserialize the list properly:
val value:List<GenreDTO> = mapper.readValue(json, object : TypeReference<List<GenreDTO>>() {})
As #JaysonMinard has pointed out you can use jackson-module-kotlin to simplify the invocation to:
val genres: List<GenreDTO> = mapper.readValue(json)
// or
val genres = mapper.readValue<List<GenreDTO>>(json)
This is possible because of reified type parameters. Consider looking at Extensions to find out details.
Following code works well for me:
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.kotlin.readValue
import com.fasterxml.jackson.module.kotlin.registerKotlinModule
val json = """[ {
"id" : "1",
"name" : "Blues"
}, {
"id" : "0",
"name" : "Rock"
} ]"""
data class GenreDTO(val id: Int, val name: String)
val mapper = ObjectMapper().registerKotlinModule()
fun main(args: Array<String>) {
val obj: List<GenreDTO> = mapper.readValue(json)
obj.forEach {
println(it)
}
}
This work because of extension function defined inside jackson-kotlin-module (that used reified generics):
public inline fun <reified T: Any> ObjectMapper.readValue(content: String): T = readValue(content, object: TypeReference<T>() {})
Thanks #JaysonMinard for notify me about it.
Output:
GenreDTO(id=1, name=Blues)
GenreDTO(id=0, name=Rock)

Spray-json convert part of the json to Map[String,List[Double]]

I have the following json
{
"op":{
"samples":{
"ep_mem_high_wat":[ 0,0,0,0,0,0,0],
"ep_mem_low_wat":[ 0,0,0,0,0,0,0]
},
"samplesCount":60,
"isPersistent":true,
"lastTStamp":1415619627689,
"interval":1000
},
"hot_keys":[
{
"name":"counter::F03E91E2A4B9C25F",
"ops":0.11010372549516878
}
]
}
I would like to parse "samples" property of this Json as Map[String,List[Double]] as
Map[String,List[Double]]("ep_mem_high_wat" -> [ 0,0,0,0,0,0,0],"ep_mem_low_wat" -> [0,0,0,0,0,0,0])
For this purpose I perform the following: I create my case classes
case class Samples(properties:Map[String,List[Double]])
case class Op(samples:Samples,samplesCount:Int,isPersistent:Boolean,lastTStamp:Long,interval:Int)
case class Key(name:String,ops:Double)
case class BucketStatisticResponse(op:Op,hot_keys:List[Key])
and then I create custom Json protocol
object BucketStatisticJsonProtocol extends DefaultJsonProtocol {
implicit object SamplesJsonFormat extends RootJsonFormat[Samples] {
def write(obj: Samples) = obj.toJson
def read(value: JsValue) = {
val props = value.asJsObject.fields
Samples(value.convertTo[Map[String,List[Double]]])
}
}
implicit val keyFormat = jsonFormat2(Key)
implicit val opFormat = jsonFormat5(Op)
implicit val bucketStatisticFormat= jsonFormat2(BucketStatisticResponse)
}
And then I'm trying to parse json
import BucketStatisticJsonProtocol._
val res = json.toJson.convertTo[BucketStatisticResponse]
As the result I'm gettign the folloing exception:
Exception in thread "main" spray.json.DeserializationException: Object expected in field 'op'
at spray.json.package$.deserializationError(package.scala:23)
at spray.json.ProductFormats$class.fromField(ProductFormats.scala:54)
at high.availability.poc.DynamicJson$BucketStatisticJsonProtocol$.fromField(DynamicJson.scala:23)
What am I doing wrong?
Look at the error message: It's complaining about the type of value it's finding in the AST for the op field. So the document you're actually parsing probably doesn't match your sample above.
Other than that, your code looks OK to me. Just as a matter of taste, I typically don't extend DefaultJsonProtocol; I put my JSON formats in the package object, and import DefaultJsonProtocol._ to handle any of the simplistic conversions as necessary.