kotlin: nested polymorphic serialization with generics - json

I want to serialize Map<String, Any> and one of the values type is Pair<Int, Int>.
How to register the Pair as polymorphic subclass for that?
val module = SerializersModule {
polymorphic(Any::class) {
subclass(Int::class, PolymorphicPrimitiveSerializer(Int.serializer()))
subclass(String::class, PolymorphicPrimitiveSerializer(String.serializer()))
subclass(Pair::class, PolymorphicSerializer(Pair::class))
}
}
val format = Json { serializersModule = module }
val mm = mapOf<String, Any>()
.plus("int-int pair") to (5 to 10))
val jsoned = format.encodeToString(mm)
val mmDecoded = format.decodeFromString(jsoned)
require(mm==mmDecoded)
should encode to json like:
[{"first": "int-int pair",
"second":{"type": "Pair", "value":
{"first": {"type": Int, "value":5}, "second": {"type":Int, "value": 10}}}}]
But produce the following error:
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: java.lang.IllegalArgumentException: Serializer for Pair
can't be registered as a subclass for polymorphic serialization
because its kind OPEN is not concrete. To work with multiple
hierarchies, register it as a base class. at
kotlinx.serialization.json.internal.PolymorphismValidator.checkKind(PolymorphismValidator.kt:41)
at
kotlinx.serialization.json.internal.PolymorphismValidator.polymorphic(PolymorphismValidator.kt:31)
at
kotlinx.serialization.modules.SerialModuleImpl.dumpTo(SerializersModule.kt:189)
at
kotlinx.serialization.json.JsonImpl.validateConfiguration(Json.kt:358)
at kotlinx.serialization.json.JsonImpl.(Json.kt:352) at
kotlinx.serialization.json.JsonKt.Json(Json.kt:189) at
kotlinx.serialization.json.JsonKt.Json$default(Json.kt:185) at
MainKt.(Main.kt:143)

I could not make it work with polymorphic subclasses, but I don't think that feature is intended for use with Any and primitives anyway (see this question). A custom serializer seems like a more appropriate and simpler solution and unlike the polymorphic serialization, it doesn't require too much custom serializer code:
#ExperimentalSerializationApi
class DynamicLookupSerializer: KSerializer<Any> {
override val descriptor: SerialDescriptor = ContextualSerializer(Any::class, null, emptyArray()).descriptor
#OptIn(InternalSerializationApi::class)
override fun serialize(encoder: Encoder, value: Any) {
val actualSerializer = encoder.serializersModule.getContextual(value::class) ?: value::class.serializer()
encoder.encodeSerializableValue(actualSerializer as KSerializer<Any>, value)
}
override fun deserialize(decoder: Decoder): Any {
return try {
PairSerializer(Int.serializer(), Int.serializer()).deserialize(decoder)
} catch (e: Throwable) {
try {
decoder.decodeInt()
} catch (e: Throwable) {
decoder.decodeString()
}
}
}
}
val module = SerializersModule {
contextual(Any::class, DynamicLookupSerializer())
contextual(Pair::class) {
PairSerializer(Int.serializer(), Int.serializer())
}
}
val format = Json { serializersModule = module }
val mm = mapOf<String, Any>()
.plus("int-int pair" to (5 to 10))
.plus("int" to 6)
.plus("string" to "some string")
.plus("another-int" to 86248726)
.plus("another-pair" to (56 to 961))
val jsoned = format.encodeToString(mm)
println(jsoned)
val mmDecoded = format.decodeFromString<Map<String, Any>>(jsoned)
require(mm==mmDecoded)
In this custom serializer, we find the actual serializer for value: Any when serializing by looking it up via its class (value::class). As a result, PairSerializer(Int.serializer(), Int.serializer()) has to be registered too so that it will be found in DynamicLookupSerializer.serialize. When deserializing, we try the supported serializers one by one (string, int, and pair of ints).
I do realize that this is not the nicest solution due to the try-catches but it does work and it's simple enough.

I solved it by providing custom Polymorphic serializer similar to how I serialized the map:
https://github.com/assafshouval/PolymorphicMapSerializer/blob/master/src/main/kotlin/Main.kt

import kotlinx.serialization.builtins.*
val pairAnyAnySerializer = PairSerializer(
PolymorphicSerializer(Any::class), PolymorphicSerializer(Any::class)
) as <KSerializer<Pair<*,*>>>()
and for every type that is defined in the serializers module for polymorphic serializatin of Any it will serialize/deserialze correct.
val json = Json {
serializersModule = SerializersModule {
polymorphic(Any::class) {
subclass(String::class, PolymorphicPrimitiveSerializer(String.serializer()))
subclass(Int::class, PolymorphicPrimitiveSerializer(Int.serializer()))
}
}
}

Related

Kotlinx Deserialization Object or List of Object to List of Object

I am currently implementing an API client with Ktor. The API I am requesting does not return a consistent JSON format.
for Example:
sometimes the JSON looks like this:
{
"description": {
"lang": "en",
"value": "an English description..."
},
...
}
and sometimes like this:
{
"description": [
{
"lang": "en",
"value": "an English description..."
},
{
"lang": "fr",
"value": "a French description..."
}
],
...
}
Now my Question:
How can I implement a Custom Kotlinx Deserializer to Decode an Object of T or a List<T> to a List<T>
My classes look like this:
#Serializable
class ResourceResponse(
#SerialName("description")
val descriptions: List<Description>
) {
#Serializable
data class Description(
#SerialName("value")
val value: String,
#SerialName("lang")
val language: String,
)
}
I want that a Json with only one Description-Object will be deserialized to a List with one Object and not specifically for the description, but in general for classes.
I've found nothing really helpful in the Web.
One solution is to first deserialize it to JsonElement, introspect and then decide how to deserialize it further into ResourceResponse:
fun decode(s: String): ResourceResponse {
val json = Json.parseToJsonElement(s).jsonObject
return when (val desc = json["description"]) {
is JsonArray -> Json.decodeFromJsonElement(json)
is JsonObject -> {
val json2 = json.toMutableMap()
json2["description"] = JsonArray(listOf(desc))
Json.decodeFromJsonElement(JsonObject(json2))
}
else -> throw IllegalArgumentException("Invalid value for \"description\": $desc")
}
}
This solution is definitely not ideal. It may be potentially less performant as we need to deserialize the whole tree into the tree of JsonElement objects only to transform it to the final types (although, maybe the library does this internally anyway). It works only for json and it is tricky to use this solution if ResourceResponse is somewhere deep into the data structure.
You can use a JsonContentPolymorphicSerializer to choose a deserializer based on the form of the JSON.
This one should work:
#Suppress("UNCHECKED_CAST")
class DescriptionsSerializer : JsonContentPolymorphicSerializer<List<ResourceResponse.Description>>(
List::class as KClass<List<ResourceResponse.Description>>
) {
// Here we check the form of the JSON we are decoding, and choose
// the serializer accordingly
override fun selectDeserializer(element: JsonElement): DeserializationStrategy<out List<ResourceResponse.Description>> {
return if (element is JsonArray)
ListSerializer(ResourceResponse.Description.serializer())
else
SingleDescriptionAsList()
}
class SingleDescriptionAsList : KSerializer<List<ResourceResponse.Description>> {
override val descriptor: SerialDescriptor
get() = ResourceResponse.Description.serializer().descriptor
override fun deserialize(decoder: Decoder): List<ResourceResponse.Description> {
return listOf(ResourceResponse.Description.serializer().deserialize(decoder))
}
override fun serialize(encoder: Encoder, value: List<ResourceResponse.Description>) {
throw Exception("Not in use")
}
}
}
You must also amend your original class to tell it to use this serializer:
#Serializable
class ResourceResponse(
#SerialName("description")
#Serializable(with = DescriptionsSerializer::class) val descriptions: List<Description>
) {
#Serializable
data class Description(
#SerialName("value")
val value: String,
#SerialName("lang")
val language: String,
)
}
Then you will be able to decode JSON objects with the single key "descriptions" using the ResourceResponse serializer.
For avoidance of doubt, if there are other keys in the JSON (it's not entirely clear from the question) then those should also be written into ResourceResponse definition.
After my research, I have now come up with a solution. For this you need a wrapper class. (here GenericResponse). I hope I can help others who have the same problem.
This is the Wrapper-Class
#Serializable(with = ListOrObjectSerializer::class)
class GenericResponse<T>(
val data: List<T> = emptyList()
) {
private var _isNothing : Boolean = false
val isNothing: Boolean
get() {
return this._isNothing
}
companion object {
fun <T> nothing(): GenericResponse<T> {
val o = GenericResponse(emptyList<T>())
o._isNothing = true
return o
}
}
}
And the Serializer looks like:
import kotlinx.serialization.KSerializer
import kotlinx.serialization.builtins.ListSerializer
import kotlinx.serialization.descriptors.SerialDescriptor
import kotlinx.serialization.encoding.Decoder
import kotlinx.serialization.encoding.Encoder
import kotlinx.serialization.json.*
class ListOrObjectSerializer<T : Any>(private val tSerializer: KSerializer<T>): KSerializer<GenericResponse<T>> {
override val descriptor: SerialDescriptor
get() = tSerializer.descriptor
override fun deserialize(decoder: Decoder): GenericResponse<T> {
val input = decoder as JsonDecoder
val jsonObj = input.decodeJsonElement()
return when(jsonObj) {
is JsonObject -> GenericResponse(listOf(Json.decodeFromJsonElement(tSerializer, jsonObj)))
is JsonArray -> GenericResponse(Json.decodeFromJsonElement(ListSerializer(tSerializer), jsonObj))
else -> return GenericResponse.nothing()
}
}
override fun serialize(encoder: Encoder, value: GenericResponse<T>) {
throw IllegalAccessError("serialize not supported")
}
}
My Data-Class look now like:
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
#Serializable
class ResourceResponse(
#SerialName("description")
val descriptions: GenericResponse<Description>? = null,
) {
#Serializable
data class Description(
#SerialName("value")
val value: String? = null,
#SerialName("lang")
val language: String? = null,
)
}
data class ResourceResponse(
#SerializedName("description") val descriptions: List<Description>,
)
data class Description(
#SerializedName("value") val value: String,
#SerializedName("lang") val language: String,
)
it should be like that

Custom KotlinX Serializer for List class

I want to make a custom List serializer that will parse invalid json arrays safely. Example: list of Int [1, "invalid_int", 2] should be parsed as [1, 2].
I've made a serializer and added it to Json provider, but serialization keeps failing after first element and cannot continue, so I'm getting list of 1 element [1], how can I handle invalid element correctly so decoder will keep parsing other elements?
class SafeListSerializerStack<E>(val elementSerializer: KSerializer<E>) : KSerializer<List<E>> {
override val descriptor: SerialDescriptor = ListSerializer(elementSerializer).descriptor
override fun serialize(encoder: Encoder, value: List<E>) {
val size = value.size
val composite = encoder.beginCollection(descriptor, size)
val iterator = value.iterator()
for (index in 0 until size) {
composite.encodeSerializableElement(descriptor, index, elementSerializer, iterator.next())
}
composite.endStructure(descriptor)
}
override fun deserialize(decoder: Decoder): List<E> {
val arrayList = arrayListOf<E>()
try {
val startIndex = arrayList.size
val messageBuilder = StringBuilder()
val compositeDecoder = decoder.beginStructure(descriptor)
while (true) {
val index = compositeDecoder.decodeElementIndex(descriptor) // fails here on number 2
if (index == CompositeDecoder.DECODE_DONE) {
break
}
try {
arrayList.add(index, compositeDecoder.decodeSerializableElement(descriptor, startIndex + index, elementSerializer))
} catch (exception: Exception) {
exception.printStackTrace() // falls here when "invalid_int" is parsed, it's ok
}
}
compositeDecoder.endStructure(descriptor)
if (messageBuilder.isNotBlank()) {
println(messageBuilder.toString())
}
} catch (exception: Exception) {
exception.printStackTrace() // falls here on number 2
}
return arrayList
}
}
Error happens after invalid element is parsed and exception is thrown at compositeDecoder.decodeElementIndex(descriptor) line with:
kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 4: Expected end of the array or comma
JSON input: [1, "invalid_int", 2]
I had a feeling that it should "swallow" invalid element and just keep moving, but instead it's stuck and cannot continue parsing, which doesn't make sense to me.
This could be done without custom serializer. Just parse everything as a String (specify isLenient = true to allow unquoted strings) and then convert to Int all valid integers:
fun main() {
val input = "[1, \"invalid_int\", 2]"
val result: List<Int> = Json { isLenient = true }
.decodeFromString<List<String>>(input)
.mapNotNull { it.toIntOrNull() }
println(result) // [1, 2]
}
In a more generic case (when the list is a field and/or its elements are not simple Ints), you'll need a custom serializer:
class SafeListSerializerStack<E>(private val elementSerializer: KSerializer<E>) : KSerializer<List<E>> {
private val listSerializer = ListSerializer(elementSerializer)
override val descriptor: SerialDescriptor = listSerializer.descriptor
override fun serialize(encoder: Encoder, value: List<E>) {
listSerializer.serialize(encoder, value)
}
override fun deserialize(decoder: Decoder): List<E> = with(decoder as JsonDecoder) {
decodeJsonElement().jsonArray.mapNotNull {
try {
json.decodeFromJsonElement(elementSerializer, it)
} catch (e: SerializationException) {
e.printStackTrace()
null
}
}
}
}
Note that this solution works only with deserialization from the Json format and requires kotlinx.serialization 1.2.0+
Found a way, we can extract json array from decoder given we are using Json to parse it
override fun deserialize(decoder: Decoder): List<E> {
val jsonInput = decoder as? JsonDecoder
?: error("Can be deserialized only by JSON")
val rawJson = jsonInput.decodeJsonElement()
if (rawJson !is JsonArray) {
return arrayListOf()
}
val jsonArray = rawJson.jsonArray
val jsonParser = jsonInput.json
val arrayList = ArrayList<E>(jsonArray.size)
jsonArray.forEach { jsonElement ->
val result = readElement(jsonParser, jsonElement)
when {
result.isSuccess -> arrayList.add(result.getOrThrow())
result.isFailure -> Log.d("ERROR", "error parsing array")
}
}
arrayList.trimToSize()
return arrayList
}
private fun readElement(json: Json, jsonElement: JsonElement): Result<E> {
return try {
Result.success(json.decodeFromJsonElement(elementSerializer, jsonElement))
} catch (exception: Exception) {
Result.failure(exception)
}
}

Same SerializedName for two different data types sometimes, but with Kotlin

The JSON file I'm pulling from unfortunately has a node with the same variable name but could have two different data types randomly. When I make a network call (using gson) I get the error:
com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected a BEGIN_ARRAY but was int at line 1 column 5344 path $[1].medium
the JSON looks like
{
"title": "Live JSON generator",
"url": google.com,
"medium": ["chicken", "radio", "room"]
}
//However sometimes medium can be:
"medium": 259
My Serialized class looks like:
data class SearchItem(
#SerializedName("title") var title: String,
#SerializedName("url") var urlStr: String,
#SerializedName("medium") val medium: List<String>? = null
) : Serializable {}
The way I'm making the network call is like this:
private val api: P1Api
fun onItemClicked(searchItem: SearchItem) {
api.getCollections { response, error ->
response.toString()
val searchItems: List<SearchItem> = Util.gson?.fromJson<List<SearchItem>>(
response.get("results").toString()
, object : TypeToken<List<SearchItem>>() {}.type)?.toList()!!
...
doStuffWithSearchItems(searchItems)
}
How do I handle both cases where "medium" can either be an array of strings or it could be an Int?
You could write custom JsonDeserializer for this case:
class SearchItemCustomDeserializer: JsonDeserializer<SearchItem> {
override fun deserialize(json: JsonElement, typeOfT: Type, context: JsonDeserializationContext): SearchItem {
val obj = json.asJsonObject
val title = obj.get("title").asString
val url = obj.get("url").asString
val mediumProp = obj.get("medium")
val medium = if(mediumProp.isJsonArray) {
mediumProp.asJsonArray.map { it.asString }
} else {
listOf(mediumProp.asString)
}
return SearchItem(
title = title,
urlStr = url,
medium = medium
)
}
}
With this class you "manually" deserialize json to object. For medium property we check is this array or simple json primitive with function mediumProp.isJsonArray. And if answer is yes - then deserialize field as json array of strings mediumProp.asJsonArray.map { it.asString } Else deserialize the field as string.
And then we register our custom SearchItemCustomDeserializer on GsonBuilder using method registerTypeAdapter
val gson = GsonBuilder()
.registerTypeAdapter(SearchItem::class.java, SearchItemCustomDeserializer())
.create()
And after this you can use this gson instance to deserialize yours objects

How to deserialize JSON with dynamic object?

I have a simple json, but the containing field has dynamic object. For instance, json can look like
{
"fixedField1": "value1",
"dynamicField1": {
"f1": "abc",
"f2": 123
}
}
or
{
"fixedField1": "value2",
"dynamicField1": {
"g1": "abc",
"g2": { "h1": "valueh1"}
}
}
I am trying to serialize this object, but not sure how to map the dynamic field
#Serializable
data class Response(
#SerialName("fixedField1")
val fixedField: String,
#SerialName("dynamicField1")
val dynamicField: Map<String, Any> // ???? what should be the type?
)
Above code fails with following error
Backend Internal error: Exception during code generation Cause:
Back-end (JVM) Internal error: Serializer for element of type Any has
not been found.
I ran into a similar problem when I had to serialize arbitrary Map<String, Any?>
The only way I managed to do this so far was to use the JsonObject/JsonElement API and combining it with the #ImplicitReflectionSerializer
The major downside is the use of reflection which will only work properly in JVM and is not a good solution for kotlin-multiplatform.
#ImplicitReflectionSerializer
fun Map<*, *>.toJsonObject(): JsonObject = JsonObject(map {
it.key.toString() to it.value.toJsonElement()
}.toMap())
#ImplicitReflectionSerializer
fun Any?.toJsonElement(): JsonElement = when (this) {
null -> JsonNull
is Number -> JsonPrimitive(this)
is String -> JsonPrimitive(this)
is Boolean -> JsonPrimitive(this)
is Map<*, *> -> this.toJsonObject()
is Iterable<*> -> JsonArray(this.map { it.toJsonElement() })
is Array<*> -> JsonArray(this.map { it.toJsonElement() })
else -> {
//supporting classes that declare serializers
val jsonParser = Json(JsonConfiguration.Stable)
val serializer = jsonParser.context.getContextualOrDefault(this)
jsonParser.toJson(serializer, this)
}
}
Then, to serialize you would use:
val response = mapOf(
"fixedField1" to "value1",
"dynamicField1" to mapOf (
"f1" to "abc",
"f2" to 123
)
)
val serialized = Json.stringify(JsonObjectSerializer, response.toJsonObject())
Note
This reflection based serialization is only necessary if you are constrained to use Map<String, Any?>
If you are free to use your own DSL to build the responses, then you can use the json DSL directly, which is very similar to mapOf
val response1 = json {
"fixedField1" to "value1",
"dynamicField1" to json (
"f1" to "abc",
"f2" to 123
)
}
val serialized1 = Json.stringify(JsonObjectSerializer, response1)
val response 2 = json {
"fixedField1" to "value2",
"dynamicField1" to json {
"g1" to "abc",
"g2" to json { "h1" to "valueh1"}
}
}
val serialized2 = Json.stringify(JsonObjectSerializer, response2)
If, however you are constrained to define a data type, and do serialization as well as deserialization you probably can't use the json DSL so you'll have to define a #Serializer using the above methods.
An example of such a serializer, under Apache 2 license, is here: ArbitraryMapSerializer.kt
Then you can use it on classes that have arbitrary Maps. In your example it would be:
#Serializable
data class Response(
#SerialName("fixedField1")
val fixedField: String,
#SerialName("dynamicField1")
#Serializable(with = ArbitraryMapSerializer::class)
val dynamicField: Map<String, Any>
)

Moshi/Kotlin - How to serialize NULL JSON strings into empty strings instead?

I'm trying to write a null-safe String adapter that will serialize this JSON {"nullString": null} into this: Model(nullString = "") so that any JSON with a 'null' value that I expect to be a String will be replaced with "" (assuming there exists a data class like this: data class Model(val nullString: String))
I wrote a custom adapter to try and handle this:
class NullStringAdapter: JsonAdapter<String>() {
#FromJson
override fun fromJson(reader: JsonReader?): String {
if (reader == null) {
return ""
}
return if (reader.peek() == NULL) "" else reader.nextString()
}
#ToJson
override fun toJson(writer: JsonWriter?, value: String?) {
writer?.value(value)
}
}
...in an attempt to solve this parsing error:
com.squareup.moshi.JsonDataException: Expected a name but was NULL at path $.nullString
Moshi parsing code:
val json = "{\"nullString\": null}"
val moshi = Moshi.Builder()
.add(KotlinJsonAdapterFactory())
.add(NullStringAdapter())
.build()
val result = moshi.adapter(Model::class.java).fromJson(configStr)
What am I missing here? Still new to moshi so any help is appreciated!
The immediate problem is the missing reader.nextNull() call to consume the null value.
There are a couple other cleanup things you can do here, too.
With #FromJson, implementing JsonAdapter is unnecessary.
Also, the JsonReader and JsonWriter are not nullable.
object NULL_TO_EMPTY_STRING_ADAPTER {
#FromJson fun fromJson(reader: JsonReader): String {
if (reader.peek() != JsonReader.Token.NULL) {
return reader.nextString()
}
reader.nextNull<Unit>()
return ""
}
}
and use add the adapter:
val moshi = Moshi.Builder()
.add(NULL_TO_EMPTY_STRING_ADAPTER)
.add(KotlinJsonAdapterFactory())
.build()