I am writing a custom Serializer for kotlin.serialization which interprets a JSON variable as a String no matter of its type. Deserializing works out fine. But I have trouble serializing the String into its original structure again.
At the moment I just deserialize the value as a String. But that adds extra quotes around the value and leads to problems.
Here is a quick example:
//
//Custom serializer:
//
object JsonToStringSerializer : KSerializer<String> {
override val descriptor: SerialDescriptor = PrimitiveDescriptor("JsonToStringSerializer", PrimitiveKind.STRING)
override fun serialize(encoder: Encoder, value: String) {
if(value.startsWith("[") || value.startsWith("{")) {
encoder.encodeString(value) //<-- bad solution
println("Oh no! Object has been turned into a String!")
}
else //will also encode Int, Float, Boolean, ... into String - but that is ok
encoder.encodeString(value)
}
override fun deserialize(decoder: Decoder): String {
val output = (decoder as JsonInput).decodeJson().toString()
return if(output.startsWith("\""))
output.substring(1, output.length-1)
else output
}
}
//
//Data structure for test:
//
#Serializable
class Test (
#Serializable(with = JsonToStringSerializer::class) val objValue: String,
val stringValue: String
)
//
//test logic:
//
fun test() {
val jsonString = "{\"objValue\": [1,2], \"stringValue\": \"test\"}"
val obj = Json(JsonConfiguration.Stable).parse(Test.serializer(), json)
val newJsonString = Json(JsonConfiguration.Stable).stringify(Test.serializer(), obj)
println(newJsonString)
//expected result: {"objValue": [1,2], "stringValue": "test"}
//actual result: {"objValue": "[1,2]", "stringValue": "test"}
}
After some digging I found that encoder in serialize() belongs to the class StreamingJsonOutput.
I can think of three possible solutions. But I ran into a wall with each of them:
If I had a way to directly print() to composer from StreamingJsonOutput I could add the String manually to the output. But unfortunately composer is private (and StreamingJsonOutput is internal as well).
Another way would be to use the configuration unquotedPrint which would create Strings without quotes. But then stringValue would not have quotes either. So I need a way to just change the configuration only temporariy.
I could also just deserialize the value String from serialize() into a real object and then just let the encoder serialize it correctly. The problem, apart from the performance overhead, is that I don't know which datastructure to expect in value. So I am not able to deserialize it.
I hope somebody has an idea how to deal with that.
Thank you very much! :)
Related
first of all, sorry for the bad english.
I'm having a little issue with the app for my company, I started learning Kotlin a few months ago, so it's everything pretty new to me, i did a little digging for most of my problems but this one I didn't find anywhere.
We have a server provind data with a Joomla API, the problem is, when I use retrofit2 to get the data with a query, it is possible to return a BEGIN_OBJECT when no data is found, and a BEGIN_ARRAY when the data is found. I found a lot of places telling when it's one but is expected another, but the two in the same response, not yet.
This is the response when the data is not found:
{"err_msg":"Produto n\u00e3o encontrado (CALOTA CORSA)","err_code":404,"response_id":"","api":"","version":"1.0","data":{}}
I called the data class for this piece ProductList, and the data I called Product, for future reference...
This is the response when data is found:
{"err_msg":"","err_code":"","response_id":522,"api":"app.produto","version":"1.0","data":[{"codigo":"0340008","filial":"CPS","referencia":"7898314110118","ncm":"38249941","codigosecundario":"146","nome":"WHITE LUB SUPER AEROSSOL 300ML 146","similar":"0012861","parceiro":"","produtosrelacionados":"0012861;0125121;0125945;0340008;0340035;0340169;0343394;0582033;0582954;0610250;1203682;1227480;1227569;1366196;1366761;1450241;1450861","marca":"ORBI QUIMICA","linha":"DESENGRIPANTE","lancamento":"2011-07-28 00:00:00","quantidadeembalagem":"12","unidademedida":"PC","caracteristicas":"OLEO WHITE LUB SUPER AEROSSOL 300ML - DESENGRIPANTE\/ LUBRIFICANTE\/ PROTETIVO 146","lado":"N\/A","ultima_atualizacao_preco":"2022-08-05 10:32:53","valor":"9.99","ultima_atualizacao_estoque":"2022-09-01 00:03:17","estoque":"200"}]}
When the response is successful, it is possible to recieve up to 10 different products.
This is my retrofit call at the ViewModel, I'm using GsonConverterFactory with retrofit.
fun getProductByCode(code: String, token: String) {
RetrofitInstance.chgApi.listProducts(code, token).enqueue(object : Callback<ProductList> {
override fun onResponse(call: Call<ProductList>, response: Response<ProductList>) {
if (response.body()?.errCode != "") {
Log.e("Response", response.body()?.errMsg!!)
errMsg.value = response.body()?.errMsg!!
} else {
errMsg.value = ""
products.value = response.body()!!.data
}
}
override fun onFailure(call: Call<ProductList>, t: Throwable) {
Log.e("Error", t.message.toString())
}
})
}
First data class
data class ProductList(
#SerializedName("err_msg") var errMsg : String,
#SerializedName("err_code") var errCode : String,
#SerializedName("response_id") var responseId : String,
#SerializedName("api") var api : String,
#SerializedName("version") var version : String,
#SerializedName("data") var data: ArrayList<Product>
)
Second data class
#Entity(tableName = PRODUCT_DATABASE)
data class Product(
#PrimaryKey
#SerializedName("codigo" ) var codigo : String,
#SerializedName("filial" ) var filial : String,
#SerializedName("referencia" ) var referencia : String,
#SerializedName("ncm" ) var ncm : String,
#SerializedName("codigosecundario" ) var codigosecundario : String,
#SerializedName("nome" ) var nome : String,
#SerializedName("similar" ) var similar : String,
#SerializedName("parceiro" ) var parceiro : String,
#SerializedName("produtosrelacionados" ) var produtosrelacionados : String,
#SerializedName("marca" ) var marca : String,
#SerializedName("linha" ) var linha : String,
#SerializedName("lancamento" ) var lancamento : String,
#SerializedName("quantidadeembalagem" ) var quantidadeembalagem : String,
#SerializedName("unidademedida" ) var unidademedida : String,
#SerializedName("caracteristicas" ) var caracteristicas : String,
#SerializedName("lado" ) var lado : String,
#SerializedName("ultima_atualizacao_preco" ) var ultimaAtualizacaoPreco : String,
#SerializedName("valor" ) var valor : String,
#SerializedName("ultima_atualizacao_estoque" ) var ultimaAtualizacaoEstoque : String,
#SerializedName("estoque" ) var estoque : String,
var cesta : Int,
var comprar : Boolean
)
The simple way to treat would be to change my data class, changing the type of the field "data" to ArrayList< Product > or to only Product, but, as far as I know, it can't be both at the same time... Any suggestions?
Assuming that, as shown in your answer, you have two separate model classes, one for a successful response and one for an error response, and a common supertype (for example an interface Response), you could solve this with a custom JsonDeserializer 1. It should based on the members and the values of the JsonObject decide as which type the data should be deserialized. This way you can keep data: List<Product> for the ProductList response.
object ProductListResponseDeserializer : JsonDeserializer<Response> {
override fun deserialize(json: JsonElement, typeOfT: Type, context: JsonDeserializationContext): Response {
val jsonObject = json.asJsonObject
val errCode = jsonObject.getAsJsonPrimitive("err_code").asString
val errMsg = jsonObject.getAsJsonPrimitive("err_msg").asString
val responseType = if (errCode.isEmpty() && errMsg.isEmpty())
ProductList::class.java
else ProductListError::class.java
return context.deserialize(json, responseType)
}
}
(Note: Instead of duplicating the strings "err_code" and "err_msg" here and in your model classes, you could also use a single constant which is read here and used for the #SerializedName in your model classes.)
You would then have to create a GsonBuilder, register the deserializer and use Retrofit's GsonConverterFactory to use the custom Gson instance:
val gson = GsonBuilder()
.registerTypeAdapter(Response::class.java, ProductListResponseDeserializer)
.create()
Retrofit retrofit = new Retrofit.Builder()
.baseUrl(...)
.addConverterFactory(GsonConverterFactory.create(gson))
.build();
And in your callback check the class of the Response instance (whether it is a ProductList or a ProductListError).
1: In general TypeAdapter should be preferred over JsonDeserializer because it is more performant, but because here the data needs to be parsed as JsonObject anyway, there is most likely no difference.
Long story short, it took me the whole day and I found a solution here:
how to handle two different Retrofit response in Kotlin?
Just changing my Callback, Call and Response to < Any >, creating a new model and doing some treatment acording to the response. The final code:
fun searchProduct(code: String, token: String) {
RetrofitInstance.chgApi.listProducts(code.uppercase(), token).enqueue(object : Callback<Any> {
override fun onResponse(call: Call<Any>, response: Response<Any>) {
val gson = Gson()
if (response.body().toString().contains("err_msg=, err_code=")) {
productList = gson.fromJson(gson.toJson(response.body()), ProductList::class.java)
products.value = productList.data
} else {
productListError = gson.fromJson(gson.toJson(response.body()), ProductListError::class.java)
errMsg.value = productListError.errMsg
}
}
override fun onFailure(call: Call<Any>, t: Throwable) {
Log.e("Error", t.message.toString())
}
})
}
TL;DR
For a json string containing ...,field=,..., Gson keeps throwing JsonSyntaxException. What can I do?
The Case
I have to communicate with a 3rd api, Which tends to provide data like this:
{
"fieldA": "stringData",
"fieldB": "",
"fieldC": ""
}
However, In my app project, it turns out to read like this:
val jsonString = "{fieldA=stringData,fieldB=,fieldC=}"
The Problem
I tried using the standard method to deserialize it:
val jsonString = "{fieldA=stringData,fieldB=,fieldC=}"
val parseJson = Gson().fromJson(jsonString, JsonObject::class.java)
assertEquals(3, parseJson.size())
But it results in a Exception:
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unexpected value at line 1 column 28 path $.fieldB
The Solutions That Don't Work
I have tried so many solutions, none of them works. Including:
Setup a custom data class and set value to nullable
data class DataExample(
val fieldA: String?,
val fieldB: String?,
val fieldC: String?,
)
val parseToObject = Gson().fromJson(jsonString, DataExample::class.java)
Using JsonElement instead:
data class DataExample(
val fieldA: JsonElement,
val fieldB: JsonElement,
val fieldC: JsonElement,
)
val parseToObject = Gson().fromJson(jsonString, DataExample::class.java)
Applying a Deserializer:
class EmptyToNullDeserializer<T>: JsonDeserializer<T> {
override fun deserialize(
json: JsonElement, typeOfT: Type, context: JsonDeserializationContext
): T? {
if (json.isJsonPrimitive) {
json.asJsonPrimitive.also {
if (it.isString && it.asString.isEmpty()) return null
}
}
return context.deserialize(json, typeOfT)
}
}
data class DataExample(
#JsonAdapter(EmptyToNullDeserializer::class)
val fieldA: String?,
#JsonAdapter(EmptyToNullDeserializer::class)
val fieldB: String?,
#JsonAdapter(EmptyToNullDeserializer::class)
val fieldC: String?,
)
val parseToObject = Gson().fromJson(jsonString, DataExample::class.java)
or using it in GsonBuilder:
val gson = GsonBuilder()
.registerTypeAdapter(DataExample::class.java, EmptyToNullDeserializer<String>())
.create()
val parseToObject = gson.fromJson(jsonString, DataExample::class.java)
What else can I do?
It is not a valid JSON. You need to parse it by yourself. Probably this string is made by using Map::toString() method.
Here is the code to parse it into Map<String, String>
val jsonString = "{fieldA=stringData,fieldB=,fieldC=}"
val userFieldsMap = jsonString.removeSurrounding("{", "}").split(",") // split by ","
.mapNotNull { fieldString ->
val keyVal = fieldString.split("=")
// check if array contains exactly 2 items
if (keyVal.size == 2) {
keyVal[0].trim() to keyVal[1].trim() // return#mapNotNull
} else {
null // return#mapNotNull
}
}
.toMap()
It turns out that, like #frc129 and many others said, it is not an valid JSON.
The truth is however, Gson handles more situation than JSON should be, like the data below:
val jsonString = "{fieldA=stringData,fieldB=s2,fieldC=s3}"
val parseJson = Gson().fromJson(jsonString, JsonObject::class.java)
// This will NOT throw exception, even the jsonString here is not actually a JSON string.
assertEquals(3, parseJson.size())
assertEquals("stringData", parseJson["fieldA"].asString)
assertEquals("s2", parseJson["fieldB"].asString)
assertEquals("s3", parseJson["fieldC"].asString)
Further investigation indicates that -- the string mentioned here and in the question -- is more like a Map to string.
I got a bit misunderstanding with GSON dealing with Map. That should be treat as a extra handy support, but not a legal procedure. In short, it is not supposed to be transformed, and data format should be fixed. I'll go work with server and base transformation then.
Just leave a note here. If someone in the future want some quick fix to string, you may take a look at #frc129 answer; however, the ideal solution to this is to fix the data provider to provide "the correct JSON format":
val jsonString = "{\"fieldA\":\"stringData\",\"fieldB\":\"\",\"fieldC\":\"\"}"
val parseJson = Gson().fromJson(jsonString, JsonObject::class.java)
assertEquals(3, parseJson.size())
assertEquals("stringData", parseJson["fieldA"].asString)
assertEquals("", parseJson["fieldB"].asString)
assertEquals("", parseJson["fieldC"].asString)
The problem I am trying to solve is perfectly described by the following text got from this link:
For a concrete example of when this could be useful, consider an API that supports partial updates of objects. Using this API, a JSON object would be used to communicate a patch for some long-lived object. Any included property specifies that the corresponding value of the object should be updated, while the values for any omitted properties should remain unchanged. If any of the object’s properties are nullable, then a value of null being sent for a property is fundamentally different than a property that is missing, so these cases must be distinguished.
That post presents a solution but using the kotlinx.serialization library, however, I must use gson library for now.
So I am trying to implement my own solution as I didn't find anything that could suit my use case (please let me know if there is).
data class MyObject(
val fieldOne: OptionalProperty<String> = OptionalProperty.NotPresent,
val fieldTwo: OptionalProperty<String?> = OptionalProperty.NotPresent,
val fieldThree: OptionalProperty<Int> = OptionalProperty.NotPresent
)
fun main() {
val gson = GsonBuilder()
.registerTypeHierarchyAdapter(OptionalProperty::class.java, OptionalPropertyDeserializer())
.create()
val json1 = """{
"fieldOne": "some string",
"fieldTwo": "another string",
"fieldThree": 18
}
"""
println("json1 result object: ${gson.fromJson(json1, MyObject::class.java)}")
val json2 = """{
"fieldOne": "some string",
"fieldThree": 18
}
"""
println("json2 result object: ${gson.fromJson(json2, MyObject::class.java)}")
val json3 = """{
"fieldOne": "some string",
"fieldTwo": null,
"fieldThree": 18
}
"""
println("json3 result object: ${gson.fromJson(json3, MyObject::class.java)}")
}
sealed class OptionalProperty<out T> {
object NotPresent : OptionalProperty<Nothing>()
data class Present<T>(val value: T) : OptionalProperty<T>()
}
class OptionalPropertyDeserializer : JsonDeserializer<OptionalProperty<*>> {
private val gson: Gson = Gson()
override fun deserialize(
json: JsonElement?,
typeOfT: Type?,
context: JsonDeserializationContext?
): OptionalProperty<*> {
println("Inside OptionalPropertyDeserializer.deserialize json:$json")
return when {
// Is it a JsonObject? Bingo!
json?.isJsonObject == true ||
json?.isJsonPrimitive == true-> {
// Let's try to extract the type in order
// to deserialize this object
val parameterizedType = typeOfT as ParameterizedType
// Returns an Present with the value deserialized
return OptionalProperty.Present(
context?.deserialize<Any>(
json,
parameterizedType.actualTypeArguments[0]
)!!
)
}
// Wow, is it an array of objects?
json?.isJsonArray == true -> {
// First, let's try to get the array type
val parameterizedType = typeOfT as ParameterizedType
// check if the array contains a generic type too,
// for example, List<Result<T, E>>
if (parameterizedType.actualTypeArguments[0] is WildcardType) {
// In case of yes, let's try to get the type from the
// wildcard type (*)
val internalListType = (parameterizedType.actualTypeArguments[0] as WildcardType).upperBounds[0] as ParameterizedType
// Deserialize the array with the base type Any
// It will give us an array full of linkedTreeMaps (the json)
val arr = context?.deserialize<Any>(json, parameterizedType.actualTypeArguments[0]) as ArrayList<*>
// Iterate the array and
// this time, try to deserialize each member with the discovered
// wildcard type and create new array with these values
val result = arr.map { linkedTreeMap ->
val jsonElement = gson.toJsonTree(linkedTreeMap as LinkedTreeMap<*, *>).asJsonObject
return#map context.deserialize<Any>(jsonElement, internalListType.actualTypeArguments[0])
}
// Return the result inside the Ok state
return OptionalProperty.Present(result)
} else {
// Fortunately it is a simple list, like Array<String>
// Just get the type as with a JsonObject and return an Ok
return OptionalProperty.Present(
context?.deserialize<Any>(
json,
parameterizedType.actualTypeArguments[0]
)!!
)
}
}
// It is not a JsonObject or JsonArray
// Let's returns the default state NotPresent.
else -> OptionalProperty.NotPresent
}
}
}
I got most of the code for the custom deserializer from here.
This is the output when I run the main function:
Inside OptionalPropertyDeserializer.deserialize json:"some string"
Inside OptionalPropertyDeserializer.deserialize json:"another string"
Inside OptionalPropertyDeserializer.deserialize json:18
json1 result object: MyObject(fieldOne=Present(value=some string), fieldTwo=Present(value=another string), fieldThree=Present(value=18))
Inside OptionalPropertyDeserializer.deserialize json:"some string"
Inside OptionalPropertyDeserializer.deserialize json:18
json2 result object: MyObject(fieldOne=Present(value=some string), fieldTwo=my.package.OptionalProperty$NotPresent#573fd745, fieldThree=Present(value=18))
Inside OptionalPropertyDeserializer.deserialize json:"some string"
Inside OptionalPropertyDeserializer.deserialize json:18
json3 result object: MyObject(fieldOne=Present(value=some string), fieldTwo=null, fieldThree=Present(value=18))
I am testing the different options for the fieldTwo and it is almost fully working, with the exception of the 3rd json, where I would expect that fieldTwo should be fieldTwo=Present(value=null) instead of fieldTwo=null.
And I see that in this situation, the custom deserializer is not even called for fieldTwo.
Can anyone spot what I am missing here? Any tip would be very appreciated!
I ended giving up of gson and move to moshi.
I implemented this behavior based on the solution presented in this comment.
I am quite new to Kotlin but and I have successfully used Kotlin serialization on many cases - works out of the box even for nested classes, mutableLists etc. However I struggle with two dimensional arrays.
My class:
import kotlinx.serialization.*
#Serializable
data class Thing(val name:String)
#Serializable
data class Array2D(val width:Int, val height:Int,
var arrayContents:Array<Array<Thing?>> = Array(1){Array(1){null} }){
init{
arrayContents = Array(width){Array(height){null} }
}
}
And when doing this:
val a = Array2D(2, 2)
a.arrayContents[0][0] = Thing("T0")
a.arrayContents[0][1] = Thing("T1")
a.arrayContents[1][0] = Thing("T2")
a.arrayContents[1][1] = Thing("T3")
val json = Json {
allowStructuredMapKeys = true
}
val jsonString = json.encodeToString(Array2D.serializer(), a)
assertEquals(
"""
{"width":2,"height":2,"arrayContents":[[{"name":"T0"},{"name":"T1"}],[{"name":"T2"},{"name":"T3"}]]}
""".trim(),
jsonString
) // encoding is OK
val b = json.decodeFromString(deserializer = Array2D.serializer(), jsonString)
// this fails to reproduce "a" and stops at first array level
// b.arrayContents = {Thing[2][]} (array of nulls) instead of {Thing[2][2]} (array of array of Thing)
If it can encode the class to String it should decode it as well, right? Or am I missing something here? Maybe I should use custom serializer but there are not many examples that fit my case. One example is https://github.com/Kotlin/kotlinx.serialization/issues/357 but it is only one level of array.
Thanks for any help :)
Serialization/deserialization for arrays (including multi-dimensional) works out of the box.
Unexpected behavior is related to init section of your data class. It's executed after deserialization happens and overwrites data parsed from JSON.
It also happens when you create instance of Array2D manually:
val x = Array2D(1, 1, Array(1) { Array(1) { Thing("0") } })
println(x.arrayContents[0][0]) //will print null
You just need to replace init block with secondary constructor (default value for arrayContents is redundant, by the way), and you may declare arrayContents as val now:
#Serializable
data class Array2D(val width: Int, val height: Int, val arrayContents: Array<Array<Thing?>>) {
constructor(width: Int, height: Int) : this(width, height, Array(width) { Array(height) { null } })
}
Given json as follows where the structure of the payload object will vary:
{
"id": 1,
"displayName": "Success",
"payload": {
"someProperty": "example",
"someOtherProperty": {
"someNestedProperty": "example"
}
}
}
...using kotlinx.serialization how can I deserialize this into the following data class, where the value of payload should be the raw json string of the payload object.
#Serializable
data class Stub(
val id: Int,
val displayName: String,
val payload: String
)
Struggled to find a way of doing this with Serializers, but it was simple enough to implement manually using JsonElement.
val jsonObject = Json.parseToJsonElement(jsonString).jsonObject
val stub = Stub(
jsonObject["id"]!!.jsonPrimitive.int,
jsonObject["displayName"]!!.jsonPrimitive.content,
jsonObject["payload"]!!.toString()
)
There is a way to handle using JSONTransformingSerializer. It allows you to transform the json prior to deserialization. In this case from a jsonElement into a jsonPrimitive (of type String).
First create a transformer as follows:
object JsonAsStringSerializer: JsonTransformingSerializer<String>(tSerializer = String.serializer()) {
override fun transformDeserialize(element: JsonElement): JsonElement {
return JsonPrimitive(value = element.toString())
}
}
Now apply this transfer to the specific element in your data class by adding...
#Serializable(with = JsonAsStringSerializer::class)
just above the property you want to transform. Like this...
#Serializable
data class Stub(
val id: Int,
val displayName: String,
#Serializable(with = JsonAsStringSerializer::class)
val payload: String
)
The value of payload will be a string:
"{'someProperty': 'example','someOtherProperty': {'someNestedProperty':'example'}"
If you are later trying to deserialize this into different models depending on the structure, check out the JsonContentPolymorphicSerializer feature.