I have JUnit test like that:
Test fun testCategoriesLoading() {
val subscriber = TestSubscriber<List<ACategory>>()
service.categories().subscribe(subscriber)
subscriber.awaitTerminalEvent()
subscriber.assertNoErrors()
}
service is Retrofit, that uses GsonConverter to deserialize json into
data class ACategory(val id: String, val title: String, val parentId: String?, val hasChildren: Boolean)
instances.
Test is passing, even if ACategory filled with id = null, title = null etc.
So, as far as i know, gson using reflection, and kotlin lazily resolves this nullability constraints on first access.
Is there any way to force this resolve?
Some good-looking solution without direct access to fields manually? I really don't want to write every assert by hand.
You could use the new Kotlin reflection. If you have an instance of ACategory, call
ACategory::class.memberProperties
.filter { !it.returnType.isMarkedNullable }
.forEach {
assertNotNull(it.get(aCategory))
}
to access all properties that are marked as not nullable and assert they're not null. Make sure, you have the reflection lib on the classpath.
Make sure you're using M14.
We ended up with hack for data classes(only use case for us, so its ok).
Calling gsonConstructedObject.copy() reveals all exceptions
Related
Intro
I'm sending JSON messages between two backend servers that use different languages. The producing
server creates a variety of JSON messages, wrapped inside a message with metadata.
The wrapping class is Message, The consuming server has to determine which type of message its
receiving based solely on the message contents.
When I try to use a star-projection to
deserialize the message, I get an error.
Example
import kotlinx.serialization.json.Json
#Language("JSON")
val carJson = """
{
"message_type": "some message",
"data": {
"info_type": "Car",
"name": "Toyota"
}
}
""".trimIndent()
// normally I wouldn't know what the Json message would be - so the type is Message<*>
val actualCarMessage = Json.decodeFromString<Message<*>>(carJson)
Error message
Exception in thread "main" java.lang.IllegalArgumentException: Star projections in type arguments are not allowed, but Message<*>
at kotlinx.serialization.SerializersKt__SerializersKt.serializerByKTypeImpl$SerializersKt__SerializersKt(Serializers.kt:81)
at kotlinx.serialization.SerializersKt__SerializersKt.serializer(Serializers.kt:59)
at kotlinx.serialization.SerializersKt.serializer(Unknown Source)
at ExampleKt.main(example.kt:96)
at ExampleKt.main(example.kt)
Class structure
I want to deserialize JSON into a data class, Message, that has a field with a generic type.
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
#Serializable
data class Message<out DataType : SpecificInformation>(
#SerialName("message_type")
val type: String,
#SerialName("data")
val data: DataType,
)
The field is constrained by a sealed interface, SpecificInformation, with some implementations.
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.JsonClassDiscriminator
#JsonClassDiscriminator("info_type")
sealed interface SpecificInformation {
#SerialName("info_type")
val infoType: String
}
#Serializable
#SerialName("User")
data class UserInformation(
#SerialName("info_type")
override val infoType: String,
val name: String,
) : SpecificInformation
// there are more implementations...
Workaround?
This is a known
issue (kotlinx.serialization/issues/944)
,
so I'm looking for workarounds.
I have control over the JSON structure and libraries - though I have a preference for
kotlinx.serialization.
I can't change that there are two JSON objects, one is inside the other, and the discriminator is
inside the inner-class.
A custom serializer would be great. But I'd prefer to have this configured on the class or file
(with #Serializable(with = ...) or #file:UseSerializers(...)) as using a
custom SerializersModule is not as seamless.
Attempt: JsonContentPolymorphicSerializer
I've written a custom serializer, which only if it's used specifically (which is something I'd like
to avoid). It's also quite clunky, breaks if the data classes change or a new one is added, and
doesn't benefit from the sealed interface.
Can this be improved so that
It can be used generically? Json.decodeFromString<Message<*>>(carJson)
It doesn't have any hard-coded strings?
class MessageCustomSerializer : JsonContentPolymorphicSerializer<Message<*>>(Message::class) {
override fun selectDeserializer(element: JsonElement): DeserializationStrategy<out Message<*>> {
val discriminator = element
.jsonObject["data"]
?.jsonObject?.get("info_type")
?.jsonPrimitive?.contentOrNull
println("found discriminator $discriminator")
val subclassSerializer = when (discriminator?.lowercase()) {
"user" -> UserInformation.serializer()
"car" -> CarInformation.serializer()
else -> throw IllegalStateException("could not find serializer for $discriminator")
}
println("found subclassSerializer $subclassSerializer")
return Message.serializer(subclassSerializer)
}
}
fun main() {
#Language("JSON")
val carJson = """
{
"message_type": "another message",
"data": {
"info_type": "Car",
"brand": "Toyota"
}
}
""".trimIndent()
val actualCarMessage =
Json.decodeFromString(MessageCustomSerializer(), carJson)
val expectedCarMessage = Message("another message", CarInformation("Car", "Toyota"))
require(actualCarMessage == expectedCarMessage) {
println("car json parsing ❌")
}
println("car json parsing ✅")
}
#Serializable(with = ... - infinite loop
I tried applying MessageCustomSerializer directly to Message...
#Serializable(with = MessageCustomSerializer::class)
data class Message<out T : SpecificInformation>(
//...
But then I couldn't access the plugin-generated serializer, and this causes an infinite loop.
return Message.serializer(subclassSerializer) // calls 'MessageCustomSerializer', causes infinite loop
#Serializer(forClass = ...) - not generic
In addition to annotating Message with #Serializable(with = MessageCustomSerializer::class), I
tried
deriving a plugin-generated serializer:
#Serializer(forClass = Message::class)
object MessagePluginGeneratedSerializer : KSerializer<Message<*>>
But this serializer is not generic, and causes an error
java.lang.AssertionError: No such value argument slot in IrConstructorCallImpl: 0 (total=0).
Symbol: MessageCustomSerializer.<init>|-5645683436151566731[0]
at org.jetbrains.kotlin.ir.expressions.IrMemberAccessExpressionKt.throwNoSuchArgumentSlotException(IrMemberAccessExpression.kt:66)
at org.jetbrains.kotlin.ir.expressions.IrFunctionAccessExpression.putValueArgument(IrFunctionAccessExpression.kt:31)
at org.jetbrains.kotlinx.serialization.compiler.backend.ir.IrBuilderExtension$DefaultImpls.irInvoke(GeneratorHelpers.kt:210)
at org.jetbrains.kotlinx.serialization.compiler.backend.ir.SerializableCompanionIrGenerator.irInvoke(SerializableCompanionIrGenerator.kt:35)
You are asking many things here, so I will simply try to give some pointers in regards to the errors you are making which you seem to be stuck on. With those in mind, and reading the documentation I link to, I believe you should be able to resolve the rest yourself.
Polymorphic serialization
Acquaint yourself with kotlinx.serialization polymorphic serialization. When you are trying to serialize Message<*> and DataType you are trying to use polymorphic serialization.
In case you are serializing Message<*> as the root object, specifying PolymorphicSerializer explicitly (as I also posted in the bug report you link to) should work. E.g., Json.decodeFromString( PolymorphicSerializer( Message::class ), carJson ).
P.s. I'm not 100% certain what you are trying to do here is the same as in the bug report. Either way, specifying the serializer explicitely should work, whether or not it is a bug that you shouldn't be required to do so.
The message_type and info_type fields you have in Message and DataType respectively are class discriminators. You need to configure this in your Json settings, and set the correct SerialName on your concrete classes for them to work. Using a different class discriminator per hierarchy is only possible starting from kotlinx.serialization 1.3.0 using #JsonClassDiscriminator.
Overriding plugin-generated serializer
But then I couldn't access the plugin-generated serializer, and this causes an infinite loop.
#Serializable(with = ...) overrides the plugin-generated serializer. If you want to retain the plugin-generated serializer, do not apply with.
When you are serializing the object directly (as the root object), you can still pass a different serializer to use as the first parameter to encode/decode. When you want to override the serializer to use for a specific property nested somewhere in the root object, use #Serializable on the property.
Polymorphism and generic classes
The "No such value argument slot in IrConstructorCallImpl: 0" error is to be expected.
You need to do more work in case you want to specify a serializer for polymorphic generic classes.
Actually, the main problem is still that there are no reified typeargs for classes in Kotlin. But here is why this bothers me in this specific case:
Suppose you have a wrapper class Wrapper that takes in a string content and a class* type and can output an object of class type retrieved by parsing content as JSON by demand by calling the function getObj():
class Wrapper<T>(private val content: String, /*private val type: KClass<*>*/) {
fun getObj(): T {
// ?
}
}
And I want to use kotlinx.serialization. Now, you might have noticed how I put an asterisk after "class" before. Here's the reason: Yes, Wrapper has to take the target class in some way, but how? Should it be just the typearg (won't work because type erausre) or a KClass reference (won't work because I need a reified typearg)?
The thing is that as far as I know, the only way to decode a generic JSON to a serializable target class is to use Json.decodeFromString<T>(content), where T is the target type and content is the JSON string. Now, T is defined to be reified (so that the type can be processed at runtime) and can only be filled with another reified typearg or an actual class reference. I can't use another reified typearg because I am in the context of a class and a class cannot have reified typeargs. I can also not use an actual class reference because the user of the class should be able to construct it with different targets, e.g. they decide what the target is, not me.
So, how do I do this with kotlinx.serialization? Is it even possible?
Ok so no one answered the question yet, but I also posted this question in the r/Kotlin subreddit. Here it is.
I actually got an answer there (credits to u/JakeWharton), and since you might get across this StackOverflow question because you googled the same question, you might be happy to find an answer here. So here's my try to paraphrase the answer:
So, basically, kotlinx-serialization does indeed not work with KClasses. But when you think about it, you only need the KClass to determine how to serialize it. And since that is determined at compile-time when you work with KXS, you actually just need to pass the serializer (the actual strategy defining how to serialize / deserialize your class). You can obtain a serializer for every class annotated with #Serializable by invoking .serializer() on it; the result will be of the type KSerializer<T>. So, instead of having
class Wrapper<T>(private val content: String, private val type: KClass<T>)
and constructing it via
val wrapper = Wrapper("{}", Foo::class)
You can do it like this:
class Wrapper<T>(private val content: String, private val serializer: KSerializer<T>)
and then construct it like this:
val wrapper = Wrapper("{}", Foo.serializer())
(supposing Foo is annotated with #Serializable)
you can then serialize and deserialize by using the KSerializer instead of a typearg, like this:
val obj: T = Json.decodeFromString(serializer, "[Your JSON String]")
val str: String = Json.encodeToString(serializer, obj)
And that's it! Just swap out your regular (K)Class approach by KSerializer and it'll work with KXS.
Serving JSON content in Ktor as described in HTTP API - Quick Start - Ktor, as shown in the examples, works for common collections (lists, maps, etc.) and data classes. However, if I want to serialize a class that is not a data class and has fields that I want to exclude, how do I specify the fields to be serialized and their serialized names? Assume that I am using Gson, can I do it in the same way as serializing a class object using Gson directly?
Using Gson, you have a couple of options to the best of my knowledge.
1. Using Transient
If you mark a field with #Transient (transient in Java) this will be excluded from serialization:
data class Foo(
#Transient val a: Int,
val b: Int)
Here, b will be serialized and a will not.
This comes with a huge downside - almost every framework in java takes #Transient into account and sometimes you don't want it to be serialized by Gson, but you might want to persist it to the database for example (if you'd be using the same class for both). To account for this, there's another option, using #Expose.
2. Using Expose
You need to create the gson instance using the builder:
val gson = GsonBuilder()
.excludeFieldsWithoutExposeAnnotation()
.create();
Now, fields without #Expose won't be serialized:
data class Foo(
val a: Int,
#Expose val b: Int)
Again, a will not be serialized, but b will.
3. Using exclusion strategies
A more advanced method is the usage of exclusion strategies. This allows for loads of introspections on the fields. From custom annotations to the field name or type.
Again, you need to create a gson with a builder:
val gson = GsonBuilder()
.addSerializationExclusionStrategy(strategyInstance)
.create();
And you define a strategy like:
object : ExclusionStrategy() {
override fun shouldSkipField(field: FieldAttributes): Boolean {
}
override fun shouldSkipClass(clazz: Class<*>): Boolean {
}
}
inside shouldSkipField you return true when you don't want to serialize the field and false when you do. Because it receives a FieldAttributes you can get a lot of properties from the field such as name and annotations. This allows for very fine-grained control.
Lastly, you can set this strategy for deserialization as well and for both - addDeserializationExclusionStrategy and setExclusionStrategies.
I am trying to build a generic CRUD interface that receives an element, searches for an implicit CRUD implementation from an evidence and also serialize the necessary object based on the parametrized type.
The code goes as follow:
private def createElement[T <: AnyRef](element: String)
(implicit ev: ResourceManager[T], m: Manifest[T]): Response = Try {
val e = Serializer.fromJson(element, m.runtimeClass)
ev.create(e, persistence)
ResponseBuilder.newBuilder().status(202).build()
}.getOrElse(ResponseBuilder.newBuilder().status(412).build())
As you can see. I receive a String, an evidence of a ResourceManager that implements the create method and a Manifest. When trying to serialize i make sure that the T type is of AnyRef but that is not the issue.
The problem is that m.runtimeClass returns Class[_] instead of Class[T].
My question is, if manifest does do what i want, how can i do this without explicitly passing the class name or whatever?
Thank you!
Consider this example using Play's JSON API (play.api.libs.json):
case class FooJson(
// lots of other fields omitted
location: Option[LocationJson]
)
object FooJson {
implicit val writes = Json.writes[FooJson]
}
and
case class LocationJson(latitude: Double, longitude: Double)
object LocationJson {
implicit val writes = Json.writes[LocationJson]
}
If location is None, the resulting JSON won't have location field at all. This is fine and understadable. But if I wanted for some reason (say, to make my API more self-documenting), how can I explicitly output null in JSON?
{
"location": null
}
I also tried defining the field as location: LocationJson and passing option.orNull to it, but it does not work (scala.MatchError: null at play.api.libs.json.OWrites$$anon$2.writes). For non-custom types such as String or Double, this approach would produce null in JSON output.
So, while using Json.writes[FooJson] as shown above (or something equally simple, i.e. not having to write a custom Writes implementation), is there a clean way to include nulls in JSON?
What I'm asking is analogous to JsonInclude.Include.ALWAYS in the Jackson library (also Jackson's default behaviour). Similarly in Gson this would be trivial
(new GsonBuilder().serializeNulls().create()).
Play 2.4.4
Greg Methvin, a Play committer, wrote this answer to me in a related GitHub issue:
The JSON macros only support one way of encoding optional values,
which is to omit None values from the JSON. This is not a bug but
rather a limitation of the implementation. If you want to include
nulls you're unfortunately going to have to implement your own Writes.
I do think we should try to provide more configurability for the
macros though.
In this case, I'll let Play exclude this field when the value is null, even if it slightly sacrifices API consistency and self-documentability. It's still such a minor thing (in this particular API) that it doesn't warrant uglifying the code as much as a custom Writes would take for a case class with a dozen values.
I'm hoping they do make this more configurable in future Play versions.
Hello from the future.
As of Play 2.7, a fairly simple solution was introduced for automated JSON codecs. We can introduce the appropriate implicit value for JsonConfiguration in the scope for the Format/Reads/Writes. The following configuration will write nulls for empty Options instead of omitting the fields entirely.
import play.api.libs.json._
implicit val config = JsonConfiguration(optionHandlers = OptionHandlers.WritesNull)
implicit val residentWrites = Json.writes[Resident]
Reference
Here's a way to do it:
object MyWrites extends DefaultWrites{
override def OptionWrites[T](implicit fmt: Writes[T]): Writes[Option[T]] = new Writes[Option[T]] {
override def writes(o: Option[T]): JsValue = {
o match {
case Some(a) => Json.toJson(a)(fmt)
case None => JsNull
}
}
}
}
This will overwrite the default implementation which will not create an element. I used this in your sample code:
case class FooJson(
// ...
location: Option[LocationJson]
)
case class LocationJson(latitude: Double, longitude: Double)
object LocationJson {
implicit val writes = Json.writes[LocationJson]
}
implicit val fooJsonWriter: Writes[FooJson] = new Writes[FooJson] {
override def writes(o: FooJson): JsValue = {
JsObject(Seq(
"location" -> Json.toJson(o.location)
// Additional fields go here.
))
}
}
Json.toJson(FooJson(None))
And got this result res0: play.api.libs.json.JsValue = {"location":null}.
if we have null values then we have to add the option with members in case class which will resolve the issue
case class response(
name:String,
age: option[int]
)
object response {
implicit val format = Json.format[response]
}
Here the option is the answer for us. and if we are the JSON response for age is coming as null and this will handle the solution for us.