How to use #kotlinx.serialization.Serializable with java.time.Instant? - json

I'm learning how to use Ktor's HttpClient. And, I'd like it to automatically convert a JSON response into a data class. I think I have all the setup correct (pasted below), but unfortunately, java.time.Instant uses import java.io.Serializable;, which I guess isn't compatible with kotlinx's #kotlinx.serialization.Serializable.
So, how can I get Ktor to recognize Instant as serializable?
val httpClient = HttpClient(CIO) {
install(JsonFeature) {
serializer = KotlinxSerializer(Json {
prettyPrint = true
isLenient = true
})
}
}
val response: MyResponse = httpClient.get(baseUrl() + "/example/path") {
contentType(ContentType.Application.Json)
}
#kotlinx.serialization.Serializable
data class MyResponse(
val name: String,
val time: Instant // ERROR: "Serializer has not been found for type 'Instant'. To use context serializer as fallback, explicitly annotate type or property with #Contextual"
)
Sidenote: Other answers using Gson or Jackson or other serializers could be useful too since they may not have to explicitly add #Serializable

One solution is to use the kotlinx-datetime library (https://github.com/Kotlin/kotlinx-datetime), which gives you a Kotlin version of Instant with the #kotlinx.serialization.Serializable that you are looking for.
So, instead of using java.time.Instant, you could use kotlinx.datetime.Instant in MyResponse.
Note that the library is experimental, and the API is subject to change.
(Source: https://github.com/Kotlin/kotlinx-datetime#using-in-your-projects)

Another solution, if you want to keep using java.time, is to create your own serializer for java.time.Instant (note experiemental APIs are used):
import kotlinx.serialization.*
import kotlinx.serialization.descriptors.*
import kotlinx.serialization.encoding.*
import java.time.Instant
#Serializer(forClass = Instant::class)
#OptIn(ExperimentalSerializationApi::class)
object InstantSerializer : KSerializer<Instant> {
override val descriptor: SerialDescriptor = PrimitiveSerialDescriptor("Instant", PrimitiveKind.LONG)
override fun serialize(encoder: Encoder, value: Instant) = encoder.encodeLong(value.toEpochMilli())
override fun deserialize(decoder: Decoder) = Instant.ofEpochMilli(decoder.decodeLong())
}
Then you'd write the serializable class like this:
import kotlinx.serialization.*
import java.time.Instant
#Serializable
data class Event(
val name: String,
#Serializable(with=InstantSerializer::class) val instant: Instant
)
Note: There are other approaches (e.g., #UseSerializers).
With this setup, the following code:
import kotlinx.serialization.json.*
import java.time.Instant
fun main() {
val event = Event("Test Event", Instant.now())
val json = Json.encodeToString(event)
println("Encoded to JSON : $json")
println("Decoded from JSON: ${Json.decodeFromString<Event>(json)}")
}
Gives this output:
Encoded to JSON : {"name":"Test Event","instant":1648329502803}
Decoded from JSON: Event(name=Test Event, instant=2022-03-26T21:18:22.803Z)

Related

Kotlinx Deserialization Object or List of Object to List of Object

I am currently implementing an API client with Ktor. The API I am requesting does not return a consistent JSON format.
for Example:
sometimes the JSON looks like this:
{
"description": {
"lang": "en",
"value": "an English description..."
},
...
}
and sometimes like this:
{
"description": [
{
"lang": "en",
"value": "an English description..."
},
{
"lang": "fr",
"value": "a French description..."
}
],
...
}
Now my Question:
How can I implement a Custom Kotlinx Deserializer to Decode an Object of T or a List<T> to a List<T>
My classes look like this:
#Serializable
class ResourceResponse(
#SerialName("description")
val descriptions: List<Description>
) {
#Serializable
data class Description(
#SerialName("value")
val value: String,
#SerialName("lang")
val language: String,
)
}
I want that a Json with only one Description-Object will be deserialized to a List with one Object and not specifically for the description, but in general for classes.
I've found nothing really helpful in the Web.
One solution is to first deserialize it to JsonElement, introspect and then decide how to deserialize it further into ResourceResponse:
fun decode(s: String): ResourceResponse {
val json = Json.parseToJsonElement(s).jsonObject
return when (val desc = json["description"]) {
is JsonArray -> Json.decodeFromJsonElement(json)
is JsonObject -> {
val json2 = json.toMutableMap()
json2["description"] = JsonArray(listOf(desc))
Json.decodeFromJsonElement(JsonObject(json2))
}
else -> throw IllegalArgumentException("Invalid value for \"description\": $desc")
}
}
This solution is definitely not ideal. It may be potentially less performant as we need to deserialize the whole tree into the tree of JsonElement objects only to transform it to the final types (although, maybe the library does this internally anyway). It works only for json and it is tricky to use this solution if ResourceResponse is somewhere deep into the data structure.
You can use a JsonContentPolymorphicSerializer to choose a deserializer based on the form of the JSON.
This one should work:
#Suppress("UNCHECKED_CAST")
class DescriptionsSerializer : JsonContentPolymorphicSerializer<List<ResourceResponse.Description>>(
List::class as KClass<List<ResourceResponse.Description>>
) {
// Here we check the form of the JSON we are decoding, and choose
// the serializer accordingly
override fun selectDeserializer(element: JsonElement): DeserializationStrategy<out List<ResourceResponse.Description>> {
return if (element is JsonArray)
ListSerializer(ResourceResponse.Description.serializer())
else
SingleDescriptionAsList()
}
class SingleDescriptionAsList : KSerializer<List<ResourceResponse.Description>> {
override val descriptor: SerialDescriptor
get() = ResourceResponse.Description.serializer().descriptor
override fun deserialize(decoder: Decoder): List<ResourceResponse.Description> {
return listOf(ResourceResponse.Description.serializer().deserialize(decoder))
}
override fun serialize(encoder: Encoder, value: List<ResourceResponse.Description>) {
throw Exception("Not in use")
}
}
}
You must also amend your original class to tell it to use this serializer:
#Serializable
class ResourceResponse(
#SerialName("description")
#Serializable(with = DescriptionsSerializer::class) val descriptions: List<Description>
) {
#Serializable
data class Description(
#SerialName("value")
val value: String,
#SerialName("lang")
val language: String,
)
}
Then you will be able to decode JSON objects with the single key "descriptions" using the ResourceResponse serializer.
For avoidance of doubt, if there are other keys in the JSON (it's not entirely clear from the question) then those should also be written into ResourceResponse definition.
After my research, I have now come up with a solution. For this you need a wrapper class. (here GenericResponse). I hope I can help others who have the same problem.
This is the Wrapper-Class
#Serializable(with = ListOrObjectSerializer::class)
class GenericResponse<T>(
val data: List<T> = emptyList()
) {
private var _isNothing : Boolean = false
val isNothing: Boolean
get() {
return this._isNothing
}
companion object {
fun <T> nothing(): GenericResponse<T> {
val o = GenericResponse(emptyList<T>())
o._isNothing = true
return o
}
}
}
And the Serializer looks like:
import kotlinx.serialization.KSerializer
import kotlinx.serialization.builtins.ListSerializer
import kotlinx.serialization.descriptors.SerialDescriptor
import kotlinx.serialization.encoding.Decoder
import kotlinx.serialization.encoding.Encoder
import kotlinx.serialization.json.*
class ListOrObjectSerializer<T : Any>(private val tSerializer: KSerializer<T>): KSerializer<GenericResponse<T>> {
override val descriptor: SerialDescriptor
get() = tSerializer.descriptor
override fun deserialize(decoder: Decoder): GenericResponse<T> {
val input = decoder as JsonDecoder
val jsonObj = input.decodeJsonElement()
return when(jsonObj) {
is JsonObject -> GenericResponse(listOf(Json.decodeFromJsonElement(tSerializer, jsonObj)))
is JsonArray -> GenericResponse(Json.decodeFromJsonElement(ListSerializer(tSerializer), jsonObj))
else -> return GenericResponse.nothing()
}
}
override fun serialize(encoder: Encoder, value: GenericResponse<T>) {
throw IllegalAccessError("serialize not supported")
}
}
My Data-Class look now like:
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
#Serializable
class ResourceResponse(
#SerialName("description")
val descriptions: GenericResponse<Description>? = null,
) {
#Serializable
data class Description(
#SerialName("value")
val value: String? = null,
#SerialName("lang")
val language: String? = null,
)
}
data class ResourceResponse(
#SerializedName("description") val descriptions: List<Description>,
)
data class Description(
#SerializedName("value") val value: String,
#SerializedName("lang") val language: String,
)
it should be like that

How to unmarshall json response removing unnecessary fields using Akka HTTP

I'm new to Akka HTTP and I want to get rid of unnecessary fields from a JSON response and take only the necessary fields. For example, I use this endpoint to get the response and it contains a bunch of fields. For the moment I only need 'name' and 'versions'. I would like to know how to deserialize this into a case class containing only 'name' and 'versions'. I coded the following lines to get the response as a string.
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.{HttpRequest, HttpResponse}
import akka.stream.scaladsl.{Flow, Sink, Source}
import akka.stream.{ActorMaterializer, OverflowStrategy}
import scala.concurrent.Future
import scala.concurrent.duration.DurationInt
import scala.language.postfixOps
import scala.util.{Failure, Success}
object SoftwareRegistry extends App {
implicit val system = ActorSystem("NPMRegistry")
implicit val materializer = ActorMaterializer()
import system.dispatcher
case class NPMPackage(name: String)
// reading the packages
val filename = "B:\\Scala\\NPMRegistry\\src\\main\\resources\\packages.txt"
val bufferedSource = scala.io.Source.fromFile(filename)
val listOfPackages: List[NPMPackage] = (for (line <- bufferedSource.getLines) yield {
NPMPackage(line.trim)
}).toList
bufferedSource.close()
// source
val sourceList = Source(listOfPackages)
// sink
val sink = Sink.foreach[NPMPackage] { p =>
// https request
val responseFuture: Future[HttpResponse] =
Http().singleRequest(HttpRequest(uri = s"https://registry.npmjs.org/${p.name}"))
val x = responseFuture
.flatMap(_.entity.toStrict(2 seconds))
.map(_.data.utf8String)
x.onComplete {
case Success(res) => println(res)
case Failure(_) => sys.error("Something went wrong")
}
}
// flow to slow things down and streaming sink to time-delayed operations
val bufferedFlow = Flow[NPMPackage]
.buffer(10, overflowStrategy = OverflowStrategy.backpressure)
.throttle(1, 3 seconds)
sourceList.async
.via(bufferedFlow).async
.to(sink)
.run()
}
And it prints the following output
For parsing json you need to use some library. In akka-http docs they use spray-json. Add the following dependency to your build.sbt with appropriate akkaHttpVersion.
"com.typesafe.akka" %% "akka-http-spray-json" % akkaHttpVersion
Now you need serializers and deserializers for your data. I am using a simple model, change it as needed.
trait Formatter extends DefaultJsonProtocol {
implicit object jsonFormat extends JsonFormat[Versions] {
override def read(json: JsValue): Versions = json match {
case JsObject(fields) =>
Versions(fields.keys.toList)
}
override def write(obj: Versions): JsValue = JsonParser(obj.toString)
}
implicit val formatterPackage: RootJsonFormat[Package] = jsonFormat2(Package)
case class Package(name: String, versions: Versions)
case class Versions(versions: List[String])
}
Finally sink:
//needed import with others
import spray.json._
object SoftwareRegistry extends App with Formatter {
//existing code
//---------
val sink = Sink.foreach[NPMPackage] { p =>
// https request
val responseFuture: Future[HttpResponse] =
Http().singleRequest(HttpRequest(uri = s"https://registry.npmjs.org/${p.name}"))
val packages = responseFuture
.flatMap(
_.entity
.dataBytes
.via(JsonFraming.objectScanner(Int.MaxValue))
.map(_.utf8String)
.map(_.parseJson.convertTo[Package])
.toMat(Sink.seq)(Keep.right)
.run()
)
packages.onComplete {
case Success(res) => println(res)
case Failure(_) => sys.error("Something went wrong")
}
}
//existing code
//---------
}

Accept all JSON from the request AKKA-HTTTP

I have one route:
val receiveMessage = post {
path("receive_message"){
entity(as[Receive]) {
message => {
println(message)
complete("ok")
}
}
}
}
He work with case class:
// spray (JSON marshalling)
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._
final case class Receive(notification_text: String)
// formats for unmarshalling and marshalling
implicit val itemFormat4 = jsonFormat1(Receive)
Is there a way to make a route that can receive any JSON objects?
You can type your route to receive a JsObject (from SprayJson):
entity(as[JsObject])

json Generic decoder with default values using scala with circe

I have encountered a weird situation.
I m trying to build a method that takes a type and a JSON
and build it into a case class instance and if needed auto-complete missing key values.
So far I managed to do everything separately but not altogether.
The case class with its defaults:
case class Foo(a: String = "empty String", b: Option[Int] = Some(1))
and when I do the conversion:
import io.circe.generic.extras.auto._
import io.circe.generic.extras.Configuration
import io.circe.parser.decode
implicit val customConfig: Configuration = Configuration.default.withDefaults
println(decode[Foo]("{}"))
this is the output I get:
Right(Foo(empty String,Some(1)))
and this is working as I expected
but when I put it into a generic method it required a to be an option due to the error:
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: DecodingFailure(Attempt to decode value on failed cursor, List(DownField(a)))
so I`m changing the case class to be
case class Foo(a: Option[String] = Some("empty String"), b: Option[Int] = Some(1))
and add the decoder:
object Foo{
implicit val decoder:Decoder[Foo] = deriveDecoder[Foo]
}
to the method:
import io.circe.Decoder
import io.circe.parser.decode
def convertToObj[T](jsonStr: String)(implicit decoder: Decoder[T]): T = {
decode[T](jsonStr)
match {
case Right(value) => value
case Left(error) => throw error
}
}
println(convertToObj[Foo]("{}"))
and the output is:
Foo(None,None)
so now I have lost my default values that I put and not able to use the automatic decoder as well.
How can I combine my two wishes into one approach?
You would need to do something like:
package foo.bar
import io.circe.Decoder
import io.circe.generic.extras.semiauto
import io.circe.generic.extras.Configuration
import io.circe.parser.decode
case class Foo(a: String = "empty String", b: Option[Int] = Some(1))
object Foo {
implicit val customConfig: Configuration = Configuration.default.withDefaults
implicit val decoder: Decoder[Foo] = semiauto.deriveConfiguredDecoder[Foo]
}
object TestApp extends App {
def convertToObj[T](jsonStr: String)(implicit decoder: Decoder[T]): T =
decode[T](jsonStr) match {
case Right(value) => value
case Left(error) => throw error
}
println(convertToObj[Foo]("{}"))
}
However, you can have circe automatically derive your decoder for you, so you can get away with less boilerplate:
package foo.bar
import io.circe.Decoder
import io.circe.generic.extras.auto._
import io.circe.generic.extras.Configuration
import io.circe.parser.decode
case class Foo(a: String = "empty String", b: Option[Int] = Some(1))
object TestApp extends App {
implicit val customConfig: Configuration = Configuration.default.withDefaults
def convertToObj[T](jsonStr: String)(implicit decoder: Decoder[T]): T =
decode[T](jsonStr) match {
case Right(value) => value
case Left(error) => throw error
}
println(convertToObj[Foo]("{}"))
}
Both of these examples give me output: Foo(empty String,Some(1))
NOTE:
method deriveDecoder in object semiauto is deprecated (since 0.12.0): Use deriveConfiguredDecoder

How to implement implicit Json Writes of Future object in Play Framework 2.x

I am new with play framework and i want to ask periodically to amazon about some products in order to insert them into a kafka topic, an error happens when i try to compile the code.
This is the code of the KafkaProducer:
file example.model.AmazonProducerExample
//UPDATED method with the suggestions from the users, thank you guys!
package example.utils
import jodd.lagarto.dom.{NodeSelector, LagartoDOMBuilder}
import example.model.AmazonProduct
import scala.collection.JavaConversions._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future
import play.api.libs.json._
import example.utils._
import example.producer._
object AmazonPageParser {
private val topicName = "amazonRatingsTopic"
private val producer = Producer[String](topicName)
def parse(productId: String): Future[AmazonProduct] = {
val url = s"http://www.amazon.com/dp/$productId"
HttpClient.fetchUrl(url) map {
httpResponse =>
if (httpResponse.getStatusCode == 200) {
val body = httpResponse.getResponseBody
val domBuilder = new LagartoDOMBuilder()
val doc = domBuilder.parse(body)
val responseUrl = httpResponse.getUri.toString
val nodeSelector = new NodeSelector(doc)
val title = nodeSelector.select("span#productTitle").head.getTextContent
val img = nodeSelector.select("div#main-image-container img").head.getAttribute("src")
val description = nodeSelector.select("div#feature-bullets").headOption.map(_.getHtml).mkString
val amazonProduct = AmazonProduct(productId, title, responseUrl, img, description)
println("amazonProduct is " + amazonProduct.toString)
amazonProduct
} else {
println("An error happened!")
throw new RuntimeException(s"Invalid url $url")
}
}//map
}//parse method
def main(args: Array[String]): Unit = {
//Scala Puzzlers...
AmazonPageParser.parse("0981531679").onSuccess { case amazonProduct =>
implicit val amazonFormat = Json.format[AmazonProduct]
producer.send(Json.toJson(amazonProduct).toString)
println("amazon product sent to kafka cluster..." + amazonProduct.toString)
}
}
}
file example.model.Models
package example.model
import play.api.libs.json.Json
import reactivemongo.bson.Macros
case class AmazonProduct(itemId: String, title: String, url: String, img: String, description: String)
case class AmazonRating(userId: String, productId: String, rating: Double)
case class AmazonProductAndRating(product: AmazonProduct, rating: AmazonRating)
// For MongoDB
object AmazonRating {
implicit val amazonRatingHandler = Macros.handler[AmazonRating]
implicit val amazonRatingFormat = Json.format[AmazonRating]
}
file example.utils.AmazonPageParser
The compiler returns me this error:
[error] /Users/aironman/my-recommendation-spark-engine/src/main/scala/example/producer/AmazonProducerExample.scala:25: No Json serializer found for type scala.concurrent.Future[example.model.AmazonProduct]. Try to implement an implicit Writes or Format for this type.
[error] producer.send(Json.toJson(amazonProduct).toString)
[error] ^
I have readed this post with most votes but it does not work for me.
Can anybody help me?
Writes[T] produces Json. You can't produce it directly from Future without blocking.
However you can add "callback" to this future, like this:
amazonPageParser.parse(productId).onSuccess { case amazonProduct =>
producer.send(Json.toJson(amazonProduct).toString)
}
Or with other Future methods, like map or foreach.