I have a dstream and trying to parse each string into an object using play json library.
It works on the outside but it doesn't work on the inside. It says no implicits found.
val textStream: ReceiverInputDStream[String] = streamingContext.socketTextStream("localhost",12345)
textStream.map{record=>
Json.parse(record).as[Car] //this doesn't works, it shows as: Json.parse(record).as[Car](...)
}
This is defined in the companion object of Car:
object Car {
implicit val carFormat = Json.format[Car]
}
It works when implicits are given explicitly inside and it seems a bit odd.
textStream.map{record=>
implicit val carFormat = Json.format[Car] //How do I avoid this
Json.parse(record).as[Car]
}
How do I avoid defining inside and make it work by defining the implicits outside ?
I also tried importing explicit inside the companion object but it doesn't make any difference.
Related
Intro
I'm sending JSON messages between two backend servers that use different languages. The producing
server creates a variety of JSON messages, wrapped inside a message with metadata.
The wrapping class is Message, The consuming server has to determine which type of message its
receiving based solely on the message contents.
When I try to use a star-projection to
deserialize the message, I get an error.
Example
import kotlinx.serialization.json.Json
#Language("JSON")
val carJson = """
{
"message_type": "some message",
"data": {
"info_type": "Car",
"name": "Toyota"
}
}
""".trimIndent()
// normally I wouldn't know what the Json message would be - so the type is Message<*>
val actualCarMessage = Json.decodeFromString<Message<*>>(carJson)
Error message
Exception in thread "main" java.lang.IllegalArgumentException: Star projections in type arguments are not allowed, but Message<*>
at kotlinx.serialization.SerializersKt__SerializersKt.serializerByKTypeImpl$SerializersKt__SerializersKt(Serializers.kt:81)
at kotlinx.serialization.SerializersKt__SerializersKt.serializer(Serializers.kt:59)
at kotlinx.serialization.SerializersKt.serializer(Unknown Source)
at ExampleKt.main(example.kt:96)
at ExampleKt.main(example.kt)
Class structure
I want to deserialize JSON into a data class, Message, that has a field with a generic type.
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
#Serializable
data class Message<out DataType : SpecificInformation>(
#SerialName("message_type")
val type: String,
#SerialName("data")
val data: DataType,
)
The field is constrained by a sealed interface, SpecificInformation, with some implementations.
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.JsonClassDiscriminator
#JsonClassDiscriminator("info_type")
sealed interface SpecificInformation {
#SerialName("info_type")
val infoType: String
}
#Serializable
#SerialName("User")
data class UserInformation(
#SerialName("info_type")
override val infoType: String,
val name: String,
) : SpecificInformation
// there are more implementations...
Workaround?
This is a known
issue (kotlinx.serialization/issues/944)
,
so I'm looking for workarounds.
I have control over the JSON structure and libraries - though I have a preference for
kotlinx.serialization.
I can't change that there are two JSON objects, one is inside the other, and the discriminator is
inside the inner-class.
A custom serializer would be great. But I'd prefer to have this configured on the class or file
(with #Serializable(with = ...) or #file:UseSerializers(...)) as using a
custom SerializersModule is not as seamless.
Attempt: JsonContentPolymorphicSerializer
I've written a custom serializer, which only if it's used specifically (which is something I'd like
to avoid). It's also quite clunky, breaks if the data classes change or a new one is added, and
doesn't benefit from the sealed interface.
Can this be improved so that
It can be used generically? Json.decodeFromString<Message<*>>(carJson)
It doesn't have any hard-coded strings?
class MessageCustomSerializer : JsonContentPolymorphicSerializer<Message<*>>(Message::class) {
override fun selectDeserializer(element: JsonElement): DeserializationStrategy<out Message<*>> {
val discriminator = element
.jsonObject["data"]
?.jsonObject?.get("info_type")
?.jsonPrimitive?.contentOrNull
println("found discriminator $discriminator")
val subclassSerializer = when (discriminator?.lowercase()) {
"user" -> UserInformation.serializer()
"car" -> CarInformation.serializer()
else -> throw IllegalStateException("could not find serializer for $discriminator")
}
println("found subclassSerializer $subclassSerializer")
return Message.serializer(subclassSerializer)
}
}
fun main() {
#Language("JSON")
val carJson = """
{
"message_type": "another message",
"data": {
"info_type": "Car",
"brand": "Toyota"
}
}
""".trimIndent()
val actualCarMessage =
Json.decodeFromString(MessageCustomSerializer(), carJson)
val expectedCarMessage = Message("another message", CarInformation("Car", "Toyota"))
require(actualCarMessage == expectedCarMessage) {
println("car json parsing ❌")
}
println("car json parsing ✅")
}
#Serializable(with = ... - infinite loop
I tried applying MessageCustomSerializer directly to Message...
#Serializable(with = MessageCustomSerializer::class)
data class Message<out T : SpecificInformation>(
//...
But then I couldn't access the plugin-generated serializer, and this causes an infinite loop.
return Message.serializer(subclassSerializer) // calls 'MessageCustomSerializer', causes infinite loop
#Serializer(forClass = ...) - not generic
In addition to annotating Message with #Serializable(with = MessageCustomSerializer::class), I
tried
deriving a plugin-generated serializer:
#Serializer(forClass = Message::class)
object MessagePluginGeneratedSerializer : KSerializer<Message<*>>
But this serializer is not generic, and causes an error
java.lang.AssertionError: No such value argument slot in IrConstructorCallImpl: 0 (total=0).
Symbol: MessageCustomSerializer.<init>|-5645683436151566731[0]
at org.jetbrains.kotlin.ir.expressions.IrMemberAccessExpressionKt.throwNoSuchArgumentSlotException(IrMemberAccessExpression.kt:66)
at org.jetbrains.kotlin.ir.expressions.IrFunctionAccessExpression.putValueArgument(IrFunctionAccessExpression.kt:31)
at org.jetbrains.kotlinx.serialization.compiler.backend.ir.IrBuilderExtension$DefaultImpls.irInvoke(GeneratorHelpers.kt:210)
at org.jetbrains.kotlinx.serialization.compiler.backend.ir.SerializableCompanionIrGenerator.irInvoke(SerializableCompanionIrGenerator.kt:35)
You are asking many things here, so I will simply try to give some pointers in regards to the errors you are making which you seem to be stuck on. With those in mind, and reading the documentation I link to, I believe you should be able to resolve the rest yourself.
Polymorphic serialization
Acquaint yourself with kotlinx.serialization polymorphic serialization. When you are trying to serialize Message<*> and DataType you are trying to use polymorphic serialization.
In case you are serializing Message<*> as the root object, specifying PolymorphicSerializer explicitly (as I also posted in the bug report you link to) should work. E.g., Json.decodeFromString( PolymorphicSerializer( Message::class ), carJson ).
P.s. I'm not 100% certain what you are trying to do here is the same as in the bug report. Either way, specifying the serializer explicitely should work, whether or not it is a bug that you shouldn't be required to do so.
The message_type and info_type fields you have in Message and DataType respectively are class discriminators. You need to configure this in your Json settings, and set the correct SerialName on your concrete classes for them to work. Using a different class discriminator per hierarchy is only possible starting from kotlinx.serialization 1.3.0 using #JsonClassDiscriminator.
Overriding plugin-generated serializer
But then I couldn't access the plugin-generated serializer, and this causes an infinite loop.
#Serializable(with = ...) overrides the plugin-generated serializer. If you want to retain the plugin-generated serializer, do not apply with.
When you are serializing the object directly (as the root object), you can still pass a different serializer to use as the first parameter to encode/decode. When you want to override the serializer to use for a specific property nested somewhere in the root object, use #Serializable on the property.
Polymorphism and generic classes
The "No such value argument slot in IrConstructorCallImpl: 0" error is to be expected.
You need to do more work in case you want to specify a serializer for polymorphic generic classes.
Actually, the main problem is still that there are no reified typeargs for classes in Kotlin. But here is why this bothers me in this specific case:
Suppose you have a wrapper class Wrapper that takes in a string content and a class* type and can output an object of class type retrieved by parsing content as JSON by demand by calling the function getObj():
class Wrapper<T>(private val content: String, /*private val type: KClass<*>*/) {
fun getObj(): T {
// ?
}
}
And I want to use kotlinx.serialization. Now, you might have noticed how I put an asterisk after "class" before. Here's the reason: Yes, Wrapper has to take the target class in some way, but how? Should it be just the typearg (won't work because type erausre) or a KClass reference (won't work because I need a reified typearg)?
The thing is that as far as I know, the only way to decode a generic JSON to a serializable target class is to use Json.decodeFromString<T>(content), where T is the target type and content is the JSON string. Now, T is defined to be reified (so that the type can be processed at runtime) and can only be filled with another reified typearg or an actual class reference. I can't use another reified typearg because I am in the context of a class and a class cannot have reified typeargs. I can also not use an actual class reference because the user of the class should be able to construct it with different targets, e.g. they decide what the target is, not me.
So, how do I do this with kotlinx.serialization? Is it even possible?
Ok so no one answered the question yet, but I also posted this question in the r/Kotlin subreddit. Here it is.
I actually got an answer there (credits to u/JakeWharton), and since you might get across this StackOverflow question because you googled the same question, you might be happy to find an answer here. So here's my try to paraphrase the answer:
So, basically, kotlinx-serialization does indeed not work with KClasses. But when you think about it, you only need the KClass to determine how to serialize it. And since that is determined at compile-time when you work with KXS, you actually just need to pass the serializer (the actual strategy defining how to serialize / deserialize your class). You can obtain a serializer for every class annotated with #Serializable by invoking .serializer() on it; the result will be of the type KSerializer<T>. So, instead of having
class Wrapper<T>(private val content: String, private val type: KClass<T>)
and constructing it via
val wrapper = Wrapper("{}", Foo::class)
You can do it like this:
class Wrapper<T>(private val content: String, private val serializer: KSerializer<T>)
and then construct it like this:
val wrapper = Wrapper("{}", Foo.serializer())
(supposing Foo is annotated with #Serializable)
you can then serialize and deserialize by using the KSerializer instead of a typearg, like this:
val obj: T = Json.decodeFromString(serializer, "[Your JSON String]")
val str: String = Json.encodeToString(serializer, obj)
And that's it! Just swap out your regular (K)Class approach by KSerializer and it'll work with KXS.
I have some scala code that requires the use of implicits for serializing and deserializing json.
We previously had something that worked by putting these implicit statements (simplified with dummies):
(in some class SomeClass1)
implicit val some1format = Json.format[SomeItem1]
implicit val some2format = Json.format[SomeItem2]
...
All as class-level variables. Any method within the class was then able to convert from Json just fine.
However, we are trying to move the implicit definitions of these formats to a separate object.
So we created an object (for example: SomeFormatters), which only contains these implicits:
object SomeFormatters {
implicit val some1format = Json.format[SomeItem1]
implicit val some2format = Json.format[SomeItem2]
}
When I try to import this object into SomeClass1, I get a compilation error saying that no deserializer was found for SomeItem1 or SomeItem2, even though I am importing SomeFormatters. (The IDE says the import of SomeFormatters is unused though, so I already knew something was off.)
What's the proper way to get SomeClass1 to know about the implicit definitions in SomeFormatters?
The issue was that there were no type annotations for implicit values -
Instead of:
implicit val some1format = Json.format[SomeItem1]
I needed to put:
implicit val some1format: Format[SomeItem1] = Json.format[SomeItem1]
I hava a Reads[T]. I would like to parse a Json object which is expected to be an array of T's. Is there a simple way to obtain a Reads[Seq[T]] without defining my Reads[T] as implicit? Essentially, I am looking for a function that takes Reads[T] and returns Reads[Seq[T]].
I came across Reads.TraversableReads, and thought that I can pass the implicit reader it needs explicitly, but this function also wants a CanBuildForm[...], which does not sound like fun.
There is a method for this in the Reads companion object: Reads.seq. Its parameter is usually implicit, but you can always call it explicitly if you want:
val a: Reads[T] = ...
val b: Reads[Seq[T]] = Reads.seq(a)
What do i need to do to extract the value for friends_count. i noticed that screen_name are already define in the Status object and case class. Do still require to extends Js or JsObject different
object TweetDetails extends Js { val friends_count = 'friends_count ? num }
and then pattern match it against each json object in the list of JsObjects as represented below. The symbols are confusing:
scala> val friends_count = 'friends_count ! num // I wish SO understood Scala's symbols
val twtJsonList = http(Status("username").timeline)
twtJsonList foreach {
js =>
val Status.user.screen_name(screen_name) = js
val Status.text(text) = js
val friends_counts(friends_count) = js //i cannot figure out how to extract this
println(friends_count)
println(screen_name)
println(text)
}
Normally, Scala symbols can be thought of as a unique identifier which will always be the same. Every symbol that is lexi-graphically identical refers to the exact same memory space. There's nothing else that's special about them from Scala's point of view.
However, Dispatch-Json pimps out symbols making them JSON property extractors. To see the code which is responsible for the pimping, check out the SymOp class and the rest of the JsonExtractor.scala code.
Let's write some code which solves the problem you are looking at and then analyze what's going on:
trait ExtUserProps extends UserProps with Js {
val friends_count = 'friends_count ! num
}
object ExtUser extends ExtUserProps with Js
val good_stuff = for {
item <- http(Status("username").timeline)
msg = Status.text(item)
user = Status.user(item)
screen_name = ExtUser.screen_name(user)
friend_count = ExtUser.friends_count(user)
} yield (screen_name, msg, friend_count)
The first thing that we're doing is extending the UserProps trait in the Dispatch-Twitter module to give it a friends_count extractor and then defining a ExtUser object which we can use to get access to that extractor. Because the ExtUserProps extends UserProps, which also extends Js, we get the method sym_add_operators in scope which turns our symbol 'friends_count into a SymOp case class. We then call the ! method on that SymOp which we then pass the Extractor num to, which creates an Extractor that looks for a property "friends_count" on a JSON object and then parses it as a number before returning. Quite a bit going on there for such a small bit of code.
The next part of the program is just a for-comprehension that calls out to the Twitter timeline for a user and parses it into JsObjects which represent each status item, them we apply the Status.text extractor to pull out the status message. Then we do the same to pull out the user. We then pull the screen_name and friend_count out of the user JsObject and finally we yield a Tuple3 back with all of the properties we were looking for. We're then left with a List[Tuple3[String,String,BigDecimal]] which you could then iterate on to print out or do whatever with.
I hope that clears some things up. The Dispatch library is very expressive but can be a little tough to wrap your head around as it uses a lot of Scala tricks which someone just learning Scala won't get right away. But keep plugging around and playing with, as well as looking at the tests and source code, and you'll see how to create powerful DSL's using Scala.