Custom akka persistence (Play!) JSON (de)serializer - json

I have my relevant actors' messages (de)serializable to/from Play! JSON. I'd like to use JSON (de)serializers for akka persistance system (if possbile).
In akka persistance documentation there is possibility to use our own serializers. Further more here are the instructions how to write custom serializers. Since akka.serialization.Serializer is expecting toBinary and fromBinary is there any way to use Play JSON serializers with akka persistence?
Thank you!
Best!

Where do you like to serialize data to?
I'm looking for a mongodb based akka persistence store with serialized objects using the json formats on my own. Maybe the following driver might be interesting for you as well:
https://github.com/scullxbones/akka-persistence-mongo/issues/16

Integrating play json into akka-persistence is complicated since play json uses instances of Format that are gathered via implicits. Akka provides just java.lang.Object for serialization and a java.lang.Class[_] for deserialization which makes resolving the correct implicit Format impossible.
What you could do is write a custom akka.serialization.Serializer that has a Map from Class[A] to Format[A]. This map can be used to find the correct format for a java.lang.Object / java.lang.Class[_]:
class JsonSerializer(serializers: Map[Class[_], Format[_]]) extends Serializer {
val charset: Charset = StandardCharsets.UTF_8
val identifier: Int = "play-json-serializer".##
val includeManifest: Boolean = true
def serializer[A](c: Class[_]): GenericFormat[A] = serializers.get(c) match {
case Some(format) => format.asInstanceOf[GenericFormat[A]]
case None => throw new RuntimeException("No Format available for " + c.getName)
}
def toBinary(o: AnyRef): Array[Byte] = jsonSerialize(o).getBytes(charset)
def fromBinary(bytes: Array[Byte], manifest: Option[Class[_]]): AnyRef = jsonDeserialize(bytes, manifest.get)
def jsonSerialize[A](a: A): String = {
implicit val format: GenericFormat[A] = serializer[A](a.getClass)
Json.stringify(Json.toJson(a))
}
def jsonDeserialize[A](bytes: Array[Byte], manifest: Class[_]): A = {
implicit val format: GenericFormat[A] = serializer[A](manifest)
Json.fromJson[A](Json.parse(new String(bytes, charset))).get
}
}
You can now inherit this class and pass play formats for all types that your akka-serializer should be able to (de)serialize to the constructor. This serializer must be configured in the akka configuration as described in the documentation:
class MyJsonSerializer extends JsonSerializer(Map(
Serializer[Foo], Serializer[...], ...
))
// Just a utility class for the pretty syntax above
object Serializer {
def apply[A](implicit format: Format[A], ctag: ClassTag[A]): (Class[A], Format[A]) =
(ctag.runtimeClass.asInstanceOf[Class[A]], format)
}

Related

Deserializing JSON into Serializable class with generic field - error: Star projections in type arguments are not allowed

Intro
I'm sending JSON messages between two backend servers that use different languages. The producing
server creates a variety of JSON messages, wrapped inside a message with metadata.
The wrapping class is Message, The consuming server has to determine which type of message its
receiving based solely on the message contents.
When I try to use a star-projection to
deserialize the message, I get an error.
Example
import kotlinx.serialization.json.Json
#Language("JSON")
val carJson = """
{
"message_type": "some message",
"data": {
"info_type": "Car",
"name": "Toyota"
}
}
""".trimIndent()
// normally I wouldn't know what the Json message would be - so the type is Message<*>
val actualCarMessage = Json.decodeFromString<Message<*>>(carJson)
Error message
Exception in thread "main" java.lang.IllegalArgumentException: Star projections in type arguments are not allowed, but Message<*>
at kotlinx.serialization.SerializersKt__SerializersKt.serializerByKTypeImpl$SerializersKt__SerializersKt(Serializers.kt:81)
at kotlinx.serialization.SerializersKt__SerializersKt.serializer(Serializers.kt:59)
at kotlinx.serialization.SerializersKt.serializer(Unknown Source)
at ExampleKt.main(example.kt:96)
at ExampleKt.main(example.kt)
Class structure
I want to deserialize JSON into a data class, Message, that has a field with a generic type.
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
#Serializable
data class Message<out DataType : SpecificInformation>(
#SerialName("message_type")
val type: String,
#SerialName("data")
val data: DataType,
)
The field is constrained by a sealed interface, SpecificInformation, with some implementations.
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.JsonClassDiscriminator
#JsonClassDiscriminator("info_type")
sealed interface SpecificInformation {
#SerialName("info_type")
val infoType: String
}
#Serializable
#SerialName("User")
data class UserInformation(
#SerialName("info_type")
override val infoType: String,
val name: String,
) : SpecificInformation
// there are more implementations...
Workaround?
This is a known
issue (kotlinx.serialization/issues/944)
,
so I'm looking for workarounds.
I have control over the JSON structure and libraries - though I have a preference for
kotlinx.serialization.
I can't change that there are two JSON objects, one is inside the other, and the discriminator is
inside the inner-class.
A custom serializer would be great. But I'd prefer to have this configured on the class or file
(with #Serializable(with = ...) or #file:UseSerializers(...)) as using a
custom SerializersModule is not as seamless.
Attempt: JsonContentPolymorphicSerializer
I've written a custom serializer, which only if it's used specifically (which is something I'd like
to avoid). It's also quite clunky, breaks if the data classes change or a new one is added, and
doesn't benefit from the sealed interface.
Can this be improved so that
It can be used generically? Json.decodeFromString<Message<*>>(carJson)
It doesn't have any hard-coded strings?
class MessageCustomSerializer : JsonContentPolymorphicSerializer<Message<*>>(Message::class) {
override fun selectDeserializer(element: JsonElement): DeserializationStrategy<out Message<*>> {
val discriminator = element
.jsonObject["data"]
?.jsonObject?.get("info_type")
?.jsonPrimitive?.contentOrNull
println("found discriminator $discriminator")
val subclassSerializer = when (discriminator?.lowercase()) {
"user" -> UserInformation.serializer()
"car" -> CarInformation.serializer()
else -> throw IllegalStateException("could not find serializer for $discriminator")
}
println("found subclassSerializer $subclassSerializer")
return Message.serializer(subclassSerializer)
}
}
fun main() {
#Language("JSON")
val carJson = """
{
"message_type": "another message",
"data": {
"info_type": "Car",
"brand": "Toyota"
}
}
""".trimIndent()
val actualCarMessage =
Json.decodeFromString(MessageCustomSerializer(), carJson)
val expectedCarMessage = Message("another message", CarInformation("Car", "Toyota"))
require(actualCarMessage == expectedCarMessage) {
println("car json parsing ❌")
}
println("car json parsing ✅")
}
#Serializable(with = ... - infinite loop
I tried applying MessageCustomSerializer directly to Message...
#Serializable(with = MessageCustomSerializer::class)
data class Message<out T : SpecificInformation>(
//...
But then I couldn't access the plugin-generated serializer, and this causes an infinite loop.
return Message.serializer(subclassSerializer) // calls 'MessageCustomSerializer', causes infinite loop
#Serializer(forClass = ...) - not generic
In addition to annotating Message with #Serializable(with = MessageCustomSerializer::class), I
tried
deriving a plugin-generated serializer:
#Serializer(forClass = Message::class)
object MessagePluginGeneratedSerializer : KSerializer<Message<*>>
But this serializer is not generic, and causes an error
java.lang.AssertionError: No such value argument slot in IrConstructorCallImpl: 0 (total=0).
Symbol: MessageCustomSerializer.<init>|-5645683436151566731[0]
at org.jetbrains.kotlin.ir.expressions.IrMemberAccessExpressionKt.throwNoSuchArgumentSlotException(IrMemberAccessExpression.kt:66)
at org.jetbrains.kotlin.ir.expressions.IrFunctionAccessExpression.putValueArgument(IrFunctionAccessExpression.kt:31)
at org.jetbrains.kotlinx.serialization.compiler.backend.ir.IrBuilderExtension$DefaultImpls.irInvoke(GeneratorHelpers.kt:210)
at org.jetbrains.kotlinx.serialization.compiler.backend.ir.SerializableCompanionIrGenerator.irInvoke(SerializableCompanionIrGenerator.kt:35)
You are asking many things here, so I will simply try to give some pointers in regards to the errors you are making which you seem to be stuck on. With those in mind, and reading the documentation I link to, I believe you should be able to resolve the rest yourself.
Polymorphic serialization
Acquaint yourself with kotlinx.serialization polymorphic serialization. When you are trying to serialize Message<*> and DataType you are trying to use polymorphic serialization.
In case you are serializing Message<*> as the root object, specifying PolymorphicSerializer explicitly (as I also posted in the bug report you link to) should work. E.g., Json.decodeFromString( PolymorphicSerializer( Message::class ), carJson ).
P.s. I'm not 100% certain what you are trying to do here is the same as in the bug report. Either way, specifying the serializer explicitely should work, whether or not it is a bug that you shouldn't be required to do so.
The message_type and info_type fields you have in Message and DataType respectively are class discriminators. You need to configure this in your Json settings, and set the correct SerialName on your concrete classes for them to work. Using a different class discriminator per hierarchy is only possible starting from kotlinx.serialization 1.3.0 using #JsonClassDiscriminator.
Overriding plugin-generated serializer
But then I couldn't access the plugin-generated serializer, and this causes an infinite loop.
#Serializable(with = ...) overrides the plugin-generated serializer. If you want to retain the plugin-generated serializer, do not apply with.
When you are serializing the object directly (as the root object), you can still pass a different serializer to use as the first parameter to encode/decode. When you want to override the serializer to use for a specific property nested somewhere in the root object, use #Serializable on the property.
Polymorphism and generic classes
The "No such value argument slot in IrConstructorCallImpl: 0" error is to be expected.
You need to do more work in case you want to specify a serializer for polymorphic generic classes.

Deserializing to java objects with Scala and json4s

I have HTTP client written in Scala that uses json4s/jackson to serialize and deserialize HTTP payloads. For now I was using only Scala case classes as model and everything was working fine, but now I have to communicate with third party service. They provided me with their own model but its written in Java, so now I need to deserialize jsons also to Java classes. It seams to work fine with simple classes but when class contains collections like Lists or Maps json4s has problems and sets all such fields to null.
Is there any way to handle such cases? Maybe I should use different formats (I'm using DefaultFormats + few custom ones). Example of problem with test:
import org.json4s.DefaultFormats
import org.json4s.jackson.Serialization.read
import org.scalatest.{FlatSpec, Matchers}
class JavaListTest extends FlatSpec with Matchers{
implicit val formats = DefaultFormats
"Java List" should "be deserialized properly" in {
val input = """{"list":["a", "b", "c"]}"""
val output = read[ObjectWithList](input)
output.list.size() shouldBe 3
}
}
And sample Java class:
import java.util.List;
public class ObjectWithList {
List<String> list;
}
I have also noticed that when I'll try to deserialize to Scala case class that contains java.util.List[String] type of field I'll get an exception of type: org.json4s.package$MappingException: Expected collection but got List[String]
Key for solving your issue, is composition of formatters. Basically you want to define JList formatter as list formatter composed with toJList function.
Unfortunately, json4s Formatters are extremely difficult to compose, so I used the Readers for you to get an idea. I also simplified an example, to having only java list:
import DefaultReaders._
import scala.collection.JavaConverters._
implicit def javaListReader[A: Reader]: Reader[java.util.List[A]] = new Reader[util.List[A]] {
override def read(value: JValue) = DefaultReaders.traversableReader[List, A].read(value).asJava
}
val input = """["a", "b", "c"]"""
val output = Formats.read[java.util.List[String]](parse(input))
To my knowledge json4s readers will not work with java classes out of the box, so you might either need to implement the Serializer[JList[_]] the same way, or mirror your java classes with case classes and use them inside your domain.
P.S.
Highly recommend you to switch to circe or argonaut, then you will forget about the most problems with jsons.

Cannot find JsonReader or JsonFormat for class. Multiple Formats inside a Protocol

I am using spray-json library to deserialize json objects. I want to be able to parse a DateTime object, specified in the nscala-time library, from a json string.
I want to deserialize a json string of this form:
{"updatedAt": "2015-05-18T23:55:49.033Z"}
For this I am defining my own protocol using spray-json's approach. My spray-json protocol goes as follows:
import spray.json._
// I want to de-serialize an object from this library
import com.github.nscala_time.time.Imports._
object CustomProtocol extends DefaultJsonProtocol {
// There are other implicit objects that describe how to format other
// classes before the one below. All of them are independent from each other
implicit object DateTimeFormat extends RootJsonFormat[DateTime] {
def write(dateTime: DateTime) = JsString(dateTime.toString)
def read(json: JsValue) = {
json.asJsObject.getFields("updatedAt") match {
case Seq(JsString(timeStamp)) => DateTime.parse(timeStamp)
case other => deserializationError("DateTime expected, found \n" + other)
}
}
}
My test (using scala-test) goes as follows:
//Previous imports
import CustomProtocol._
test("Can parse DateTime from Json string") {
val DateTimeJson = """ {"updatedAt": "2015-05-18T23:55:49.033Z"} """
val DateTimefromJson = DateTimeJson.parseJson.convertTo[DateTime]
val realDateTime = DateTime.parse("2015-05-18T23:55:49.033Z")
assert(DateTimefromJson.equals(realDateTime))
I get the following compilation error:
{Directory}/src/test/scala/messageJsonParserTest.scala:29: Cannot find
JsonReader or JsonFormat type class for com.github.nscala_time.time.Imports.DateTime
[error] val DateTimefromJson = DateTimeJson.parseJson.convertTo[DateTime]
Can anybody spot the error in my code?
I'm sorry if this question has been asked before. After browsing my mind and the internet for a whole day I have not been able to come up with a solution.

No Json serializer found for type scala.concurrent.Future[List[models.Product]]

I'm starting to play with Play! Framework for Scala and trying to build a JSON REST API with MongoDB.
I'm using play-mongodb-driver and trying to build the findAll method for my API.
So I have this model:
//models/product.scala
case class Product(_id: String, name: String, price: Double)
object Product {
val collection = MongoClients.create().database("shopping").collection("products")
implicit val format = Json.format[Product]
def findAll = {
collection.find().collect[Product]
}
}
And this controller:
//controllers/Product.scala
class Products #Inject()(mongo: MongoClient) extends Controller {
import models._
def listProducts = Action.async {
Json.toJson(Product.findAll)
}
}
and all I got is this error:
No Json serializer found for type scala.concurrent.Future[List[models.Product]]. Try to implement an implicit Writes or Format for this type.
How can I make this works?
I saw in some examples people implementing implicti val format = Json.format[Model] and thought this could solve the problem, but it didn't...
I don't get what is Product.findAll method but it looks like this method returns
Future[List[Product]]
The problem is that you trying to serialize Future to json, but you should serialize the result of Future like this:
def listProducts = Action.async {
Product
.findAll
.map(Json.arr(_))
.map(arr => Ok(arr(0)))
}

How to setup implicit json convertion for spray+akka actor

I'm Java developer and pretty new to scala.
I'm implementing some rest API that use spray and akka
The API should expose some kind of user CRUD. I'll use only create user in this question...
trait DefaultJsonFormats extends DefaultJsonProtocol with SprayJsonSupport with MetaMarshallers {}
class RegistrationService(registration: ActorRef)
(implicit executionContext: ExecutionContext)
extends Directives with DefaultJsonFormats {
implicit val timeout = Timeout(2.seconds)
implicit val userFormat = jsonFormat3(User)
implicit val registerFormat = jsonFormat1(Register)
implicit val registeredFormat = jsonFormat1(Registered)
val route =
path("register") {
post { handleWith { ru: Register => (registration ? ru).mapTo[Registered] } }
}
//------ Actor
object RegistrationActor {
case class User(id:String, name:String)
case class Register(user: User)
case class Registered(status: String)
case object NotRegistered
}
class RegistrationActor(implDef: String) extends Actor {
def receive: Receive = {
case Register(user)=>
val status=// create user real code with return status
sender ! new Registered(status)
} }
In this approach the json serialization and desiarelization is pretty annoying. For every object I need to deal with API I must define the appropriate format
implicit val userFormat = jsonFormat3(User)
implicit val registerFormat = jsonFormat1(Register)
implicit val registeredFormat = jsonFormat1(Registered)
I would like to avoid such definition and use some general json converter and return a pojo objects, so the conversion will happen under-the-hood
The question is how can I change this code to use by default Gson/Jackson/Spray default converter and avoid definition of the implicit ... jsonFormats?
For every object I need to deal with API I must define the appropriate format
It is normal to do this once, in a "JsonProtocol" class and import that where needed, rather than defining new formats each time:
import MyJsonProtocol._
val route =
path("register") {
post { handleWith { ru: Register => (registration ? ru).mapTo[Registered] } }
how can I change this code to use by default Gson/Jackson/Spray default converter and avoid definition of the implicit ... jsonFormats?
You would need to declare an implicit marshaller from Registered to HttpResponse (or an intermediate value like String) which was backed by Jackson instead of spray-json, then import that marshaller instead of SprayJsonSupport.
Have a look at the implementation of SprayJsonSupport to see how to do this. It's fairly straightforward, if you're comfortable with implicit conversions.
You can also see how this is done in Json4sSupport in Spray -- that trait implements a Marshaller[T, String] for ALL types T. Then, at runtime, the Json4s library will try to serialize the object to JSON.
In this approach the json serialization and desiarelization is pretty annoying
There are two main advantages of spray-jsons approach over Jackson's:
There is no reflection, so it is faster at runtime
This is no runtime determining of JSON formats, so any issues are caught at compile-time