immutable Map (de)serialization to/from Play JSON - json

I have following (simplified) structure:
case class MyKey(key: String)
case class MyValue(value: String)
Let's assume that I have Play JSON formatters for both case classes.
As an example I have:
val myNewMessage = collection.immutable.Map(MyKey("key1") -> MyValue("value1"), MyKey("key2") -> MyValue("value2"))
As a result of following transformation
play.api.libs.json.Json.toJson(myNewMessage)
I'm expecting something like:
{ "key1": "value1", "key2": "value2" }
I have tried writing the formatter, but somehow I can not succeed:
implicit lazy val mapMyKeyMyValueFormat: Format[collection.immutable.Map[MyKey, MyValue]] = new Format[collection.immutable.Map[MyKey, MyValue]] {
override def writes(obj: collection.immutable.Map[MyKey, MyValue]): JsValue = Json.toJson(obj.map {
case (key, value) ⇒ Json.toJson(key) -> Json.toJson(value)
})
override def reads(json: JsValue): JsResult[collection.immutable.Map[MyKey, MyValue]] = ???
}
I have no idea how to write proper reads function. Is there any simpler way of doing it? I'm also not satisfied with my writes function.
Thx!

The reason the writes method is not working is because you're transforming the Map[MyKey, MyValue] into a Map[JsValue, JsValue], but you can't serialize that to JSON. The JSON keys need to be strings, so you need some way of transforming MyKey to some unique String value. Otherwise you'd be trying to serialize something like this:
{"key": "keyName"} : {"value": "myValue"}
Which is not valid JSON.
If MyKey is as simple as stated in your question, this can work:
def writes(obj: Map[MyKey, MyValue]): JsValue = Json.toJson(obj.map {
case (key, value) => key.key -> Json.toJson(value)
}) // ^ must be a String
Play will then know how to serialize a Map[String, MyValue], given the appropriate Writes[MyValue].
But I'm not certain that's what you want. Because it produces this:
scala> Json.toJson(myNewMessage)
res0: play.api.libs.json.JsValue = {"key1":{"value":"value1"},"key2":{"value":"value2"}}
If this is the output you want:
{ "key1": "value1", "key2": "value2" }
Then your Writes should look more like this:
def writes(obj: Map[MyKey, MyValue]): JsValue = {
obj.foldLeft(JsObject(Nil)) { case (js, (key, value)) =>
js ++ Json.obj(key.key -> value.value)
}
}
Which produces this:
scala> writes(myNewMessage)
res5: play.api.libs.json.JsValue = {"key1":"value1","key2":"value2"}
Reads are easy so long as the structure of MyKey and MyValue are the same, otherwise I have no idea what you'd want it to do. It's very dependent on the actual structure you want. As is, I would suggest leveraging existing Reads[Map[String, String]] and transforming it to the type you want.
def reads(js: JsValue): JsResult[Map[MyKey, MyValue]] = {
js.validate[Map[String, String]].map { case kvMap =>
kvMap.map { case (key, value) => MyKey(key) -> MyValue(value) }
}
}
It's hard to see much else without knowing the actual structure of the data. In general I stay away from having to serialize and deserialize Maps.

Related

Extracting string from JSON using Json4s using Scala

I have a JSON body in the following form:
val body =
{
"a": "hello",
"b": "goodbye"
}
I want to extract the VALUE of "a" (so I want "hello") and store that in a val.
I know I should use "parse" and "Extract" (eg. val parsedjson = parse(body).extract[String]) but I don't know how to use them to specifically extract the value of "a"
To use extract you need to create a class that matches the shape of the JSON that you are parsing. Here is an example using your input data:
val body ="""
{
"a": "hello",
"b": "goodbye"
}
"""
case class Body(a: String, b: String)
import org.json4s._
import org.json4s.jackson.JsonMethods._
implicit val formats = DefaultFormats
val b = Extraction.extract[Body](parse(body))
println(b.a) // hello
You'd have to use pattern matching/extractors:
val aOpt: List[String] = for {
JObject(map) <- parse(body)
JField("a", JString(value)) <- map
} yield value
alternatively use querying DSL
parse(body) \ "a" match {
case JString(value) => Some(value)
case _ => None
}
These are options as you have no guarantee that arbitrary JSON would contain field "a".
See documentation
extract would make sense if you were extracting whole JObject into a case class.

discover type from CSV columns in scala

I want to read a generic CSV file with headers but unknown column number into a typed structure. My question is kind of the same as Strongly typed access to csv in scala? but with the fact I would have no schema to pass to the parser...
Until now, I was using Jackson CSV mapper to read each row as a Map[String,String], and it was working well.
import com.fasterxml.jackson.module.scala.DefaultScalaModule
def genericStringIterator(input: InputStream): Iterator[Map[String, String]] = {
val mapper = new CsvMapper()
mapper.registerModule(DefaultScalaModule)
val schema = CsvSchema.emptySchema.withHeader
val iterator = mapper
.readerFor(classOf[Map[String, String]])
.`with`(schema)
.readValues[Map[String, String]](input)
iterator.asScala
}
Now, we need the field to be typed, so 4.2 would be a Double but "4.2" would still be a String.
We are using play-json everywhere in our project, and so I know JsValue already has a great type inference for generic stuff like that.
As paly-json it is based on Jackson too, I thought it would be great to have something like
import play.api.libs.json.jackson.PlayJsonModule
def genericStringIterator(input: InputStream): Iterator[JsValue] = {
val mapper = new CsvMapper()
mapper.registerModule(PlayJsonModule)
val schema = CsvSchema.emptySchema.withHeader
val iterator = mapper
.readerFor(classOf[JsValue])
.`with`(schema)
.readValues[JsValue](input)
iterator.asScala
}
But when I try the former code, I get an exception :
val iterator = CSV.genericAnyIterator(input(
"""foo,bar,baz
|"toto",42,43
|"tata",,45
| titi,87,88
|"tutu",,
|""".stripMargin))
iterator
.foreach { a =>
println(a)
}
java.lang.RuntimeException: We should be reading map, something got wrong
at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:165)
at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:128)
at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:123)
at com.fasterxml.jackson.databind.MappingIterator.nextValue(MappingIterator.java:277)
at com.fasterxml.jackson.databind.MappingIterator.next(MappingIterator.java:192)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:40)
at scala.collection.Iterator.foreach(Iterator.scala:929)
at scala.collection.Iterator.foreach$(Iterator.scala:929)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
at my.company.csv.CSVSpec$$anon$4.<init>(CSVSpec.scala:240)
Is there something I'm doing wrong ?
I don't care particulary to have a play-json JsValue in the end, any Json structure with generic typed field would be ok. Is there another lib I could use for that ? For what I found, all other libs are based on a mapping given to the CSV Reader in advance, and what is important for me is to be able to infer the type from the CSV.
Ok, I was lazy to want to find something working out of the box :)
In fact it was easy to implement it myself.
I went looking to lib in other languages that does this infering (PapaParse in JS, Pandas in python), and discovered that they were doing a test-and-retry with priority to guess the types.
So I implemented it myself, and it works fine !
Here it is:
def genericAnyIterator(input: InputStream): Iterator[JsValue] = {
// the result of former code, mapping to Map[String,String]
val strings = genericStringIterator(input)
val decimalRegExp: Regex = "(\\d*){1}(\\.\\d*){0,1}".r
val jsValues: Iterator[Map[String, JsValue]] = strings.map { m =>
m.mapValues {
case "" => None
case "false" | "FALSE" => Some(JsFalse)
case "true" | "TRUE" => Some(JsTrue)
case value#decimalRegExp(g1, g2) if !value.isEmpty => Some(JsNumber(value.toDouble))
case "null" | "NULL" => Some(JsNull)
case value#_ => Some(JsString(value))
}
.filter(_._2.isDefined)
.mapValues(_.get)
}
jsValues.map(map => JsObject(map.toSeq))
}
which does in a test
it should "read any csv in JsObject" in new WithInputStream {
val iterator = CSV.genericAnyIterator(input(
"""foo,bar,baz
|"toto",NaN,false
|"tata",,TRUE
|titi,87.79,88
|"tutu",,null
|"tete",5.,.5
|""".stripMargin))
val result: Seq[JsValue] = iterator.toSeq
result should be(Stream(
Json.obj("foo" -> "toto", "bar" -> "NaN", "baz" -> false)
, Json.obj("foo" -> "tata", "baz" -> true)
, Json.obj("foo" -> "titi", "bar" -> 87.79, "baz" -> 88)
, Json.obj("foo" -> "tutu", "baz" -> JsNull)
, Json.obj("foo" -> "tete", "bar" -> 5, "baz" -> 0.5)
) )
}

Play framework (Scala) - Getting subset of json that contains arrays

I have many very large json-objects that I return from Play Framework with Scala.
In most cases the user doesn't need all the data in the objects, only a few fields. So I want to pass in the paths I need (as query parameters), and return a subset of the json object.
I have looked at using JSON Transformers for this task.
Filter code
def filterByPaths(paths: List[JsPath], inputObject: JsObject) : JsObject = {
paths
.map(_.json.pick)
.map(inputObject.transform)
.filter(_.isSuccess)
.map { case JsSuccess(value, path) => (value, path) }
.foldLeft(Json.obj()) { (obj, jsValueAndPath) =>
val(jsValue, path) = jsValueAndPath
val transformer = __.json.update(path.json.put(jsValue))
obj.transform(transformer).get
}
}
Usage:
val input = Json.obj(
"field1" -> Json.obj(
"field2" -> "right result"
),
"field4" -> Json.obj(
"field5" -> "not included"
),
)
val result = filterByPaths(List(JsPath \ "field1" \ "field2"), input)
// {"field1":{"field2":"right result"}}
Problem
This code works fine for JsObjects. But I can't make it work if there are JsArrays in the strucure. I had hoped that my JsPath could contain an index to look up the field, but that's not the case. (Don't know why I assumed that, maybe my head was too far in the JavaScript-world)
So this would fail to return the first entry in the Array:
val input: JsObject = Json.parse("""
{
"arr1" : [{
"field1" : "value1"
}]
}
""").as[JsObject]
val result = filterByPaths(List(JsPath \ "arr1" \ "0"), input)
// {}
Question
My question is: How can I return a subset of a json structure that contains arrays?
Alternative solution
I have the data as a case class first, and I serialize it to Json, and then run filterByPaths on it. Having a Reader that only creates the json I need in the first place might be a better solution, but creating a Reader on the fly, with configuration from queryparams seamed a more difficult task, then just stripping down the json afterwards.
The example of the returning array element:
val input: JsValue = Json.parse("""
{
"arr1" : [{
"field1" : "value1"
}]
}
""")
val firstElement = (input \ "arr1" \ 0).get
val firstElementAnotherWay = input("arr1")(0)
More about this in the Play Framework documentation: https://www.playframework.com/documentation/2.6.x/ScalaJson
Update
It looks like you got the old issue RuntimeException: expected KeyPathNode. JsPath.json.put, JsPath.json.update can't past an object to a nesting array.
https://github.com/playframework/playframework/issues/943
https://github.com/playframework/play-json/issues/82
What you can do:
Use the JSZipper: https://github.com/mandubian/play-json-zipper
Create a script to update arrays "manually"
If you can afford it, strip array in a resulting object
Example of stripping array (point 3):
def filterByPaths(paths: List[JsPath], inputObject: JsObject) : JsObject = {
paths
.map(_.json.pick)
.map(inputObject.transform)
.filter(_.isSuccess)
.map { case JsSuccess(value, path) => (value, path)}
.foldLeft(Json.obj()) { (obj, jsValueAndPath) =>
val (jsValue, path) = jsValueAndPath
val arrayStrippedPath = JsPath(path.path.filter(n => !(n.toJsonString matches """\[\d+\]""")))
val transformer = __.json.update(arrayStrippedPath.json.put(jsValue))
obj.transform(transformer).get
}
}
val result = filterByPaths(List(JsPath \ "arr1" \ "0"), input)
// {"arr1":{"field1":"value1"}}
The example
The best to handle JSON objects is by using case classes and create implicit Reads and Writes, by that you can handle errors every fields directly. Don't make it complicated.
Don't use .get() much recommended to use .getOrElse() because scala is a type-safe programming language.
Don't just use any Libraries except you know the process behind it, much better to create your own parsing method with simplified solution to save memory.
I hope it will help you..

Rename JSON fields with circe

I want to have different names of fields in my case classes and in my JSON, therefore I need a comfortable way of renaming in both, encoding and decoding.
Does someone have a good solution ?
You can use Custom key mappings via annotations. The most generic way is the JsonKey annotation from io.circe.generic.extras._. Example from the docs:
import io.circe.generic.extras._, io.circe.syntax._
implicit val config: Configuration = Configuration.default
#ConfiguredJsonCodec case class Bar(#JsonKey("my-int") i: Int, s: String)
Bar(13, "Qux").asJson
// res5: io.circe.Json = JObject(object[my-int -> 13,s -> "Qux"])
This requires the package circe-generic-extras.
Here's a code sample for Decoder (bit verbose since it won't remove the old field):
val pimpedDecoder = deriveDecoder[PimpClass].prepare {
_.withFocus {
_.mapObject { x =>
val value = x("old-field")
value.map(x.add("new-field", _)).getOrElse(x)
}
}
}
implicit val decodeFieldType: Decoder[FieldType] =
Decoder.forProduct5("nth", "isVLEncoded", "isSerialized", "isSigningField", "type")
(FieldType.apply)
This is a simple way if you have lots of different field names.
https://circe.github.io/circe/codecs/custom-codecs.html
You can use the mapJson function on Encoder to derive an encoder from the generic one and remap your field name.
And you can use the prepare function on Decoder to transform the JSON passed to a generic Decoder.
You could also write both from scratch, but it may be a ton of boilerplate, those solutions should both be a handful of lines max each.
The following function can be used to rename a circe's JSON field:
import io.circe._
object CirceUtil {
def renameField(json: Json, fieldToRename: String, newName: String): Json =
(for {
value <- json.hcursor.downField(fieldToRename).focus
newJson <- json.mapObject(_.add(newName, value)).hcursor.downField(fieldToRename).delete.top
} yield newJson).getOrElse(json)
}
You can use it in an Encoder like so:
implicit val circeEncoder: Encoder[YourCaseClass] = deriveEncoder[YourCaseClass].mapJson(
CirceUtil.renameField(_, "old_field_name", "new_field_name")
)
Extra
Unit tests
import io.circe.parser._
import org.specs2.mutable.Specification
class CirceUtilSpec extends Specification {
"CirceUtil" should {
"renameField" should {
"correctly rename field" in {
val json = parse("""{ "oldFieldName": 1 }""").toOption.get
val resultJson = CirceUtil.renameField(json, "oldFieldName", "newFieldName")
resultJson.hcursor.downField("oldFieldName").focus must beNone
resultJson.hcursor.downField("newFieldName").focus must beSome
}
"return unchanged json if field is not found" in {
val json = parse("""{ "oldFieldName": 1 }""").toOption.get
val resultJson = CirceUtil.renameField(json, "nonExistentField", "newFieldName")
resultJson must be equalTo json
}
}
}
}

Play Json API: Convert a JsArray to a JsResult[Seq[Element]]

I have a JsArray which contains JsValue objects representing two different types of entities - some of them represent nodes, the other part represents edges.
On the Scala side, there are already case classes named Node and Edge whose supertype is Element. The goal is to transform the JsArray (or Seq[JsValue]) to a collection that contains the Scala types, e.g. Seq[Element] (=> contains objects of type Node and Edge).
I have defined Read for the case classes:
implicit val nodeReads: Reads[Node] = // ...
implicit val edgeReads: Reads[Edge] = // ...
Apart from that, there is the first step of a Read for the JsArray itself:
implicit val elementSeqReads = Reads[Seq[Element]](json => json match {
case JsArray(elements) => ???
case _ => JsError("Invalid JSON data (not a json array)")
})
The part with the question marks is responsible for creating a JsSuccess(Seq(node1, edge1, ...) if all elements of the JsArray are valid nodes and edges or a JsError if this is not the case.
However, I'm not sure how to do this in an elegant way.
The logic to distinguish between nodes and edges could look like this:
def hasType(item: JsValue, elemType: String) =
(item \ "elemType").asOpt[String] == Some(elemType)
val result = elements.map {
case n if hasType(n, "node") => // use nodeReads
case e if hasType(e, "edge") => // use edgeReads
case _ => JsError("Invalid element type")
}
The thing is that I don't know how to deal with nodeReads / edgeReads at this point. Of course I could call their validate method directly, but then result would have the type Seq[JsResult[Element]]. So eventually I would have to check if there are any JsError objects and delegate them somehow to the top (remember: one invalid array element should lead to a JsError overall). If there are no errors, I still have to produce a JsSuccess[Seq[Element]] based on result.
Maybe it would be a better idea to avoid the calls to validate and work temporarily with Read instances instead. But I'm not sure how to "merge" all of the Read instances at the end (e.g. in simple case class mappings, you have a bunch of calls to JsPath.read (which returns Read) and in the end, validate produces one single result based on all those Read instances that were concatenated using the and keyword).
edit: A little bit more information.
First of all, I should have mentioned that the case classes Node and Edge basically have the same structure, at least for now. At the moment, the only reason for separate classes is to gain more type safety.
A JsValue of an element has the following JSON-representation:
{
"id" : "aet864t884srtv87ae",
"type" : "node", // <-- type can be 'node' or 'edge'
"name" : "rectangle",
"attributes": [],
...
}
The corresponding case class looks like this (note that the type attribute we've seen above is not an attribute of the class - instead it's represented by the type of the class -> Node).
case class Node(
id: String,
name: String,
attributes: Seq[Attribute],
...) extends Element
The Read is as follows:
implicit val nodeReads: Reads[Node] = (
(__ \ "id").read[String] and
(__ \ "name").read[String] and
(__ \ "attributes").read[Seq[Attribute]] and
....
) (Node.apply _)
everything looks the same for Edge, at least for now.
Try defining elementReads as
implicit val elementReads = new Reads[Element]{
override def reads(json: JsValue): JsResult[Element] =
json.validate(
Node.nodeReads.map(_.asInstanceOf[Element]) orElse
Edge.edgeReads.map(_.asInstanceOf[Element])
)
}
and import that in scope, Then you should be able to write
json.validate[Seq[Element]]
If the structure of your json is not enough to differentiate between Node and Edge, you could enforce it in the reads for each type.
Based on a simplified Node and Edge case class (only to avoid any unrelated code confusing the answer)
case class Edge(name: String) extends Element
case class Node(name: String) extends Element
The default reads for these case classes would be derived by
Json.reads[Edge]
Json.reads[Node]
respectively. Unfortunately since both case classes have the same structure these reads would ignore the type attribute in the json and happily translate a node json into an Edge instance or the opposite.
Lets have a look at how we could express the constraint on type all by itself :
def typeRead(`type`: String): Reads[String] = {
val isNotOfType = ValidationError(s"is not of expected type ${`type`}")
(__ \ "type").read[String].filter(isNotOfType)(_ == `type`)
}
This method builds a Reads[String] instance which will attempt to find a type string attribute in the provided json. It will then filter the JsResult using the custom validation error isNotOfType if the string parsed out of the json doesn't matched the expected type passed as argument of the method. Of course if the type attribute is not a string in the json, the Reads[String] will return an error saying that it expected a String.
Now that we have a read which can enforce the value of the type attribute in the json, all we have to do is to build a reads for each value of type which we expect and compose it with the associated case class reads. We can used Reads#flatMap for that ignoring the input since the parsed string is not useful for our case classes.
object Edge {
val edgeReads: Reads[Edge] =
Element.typeRead("edge").flatMap(_ => Json.reads[Edge])
}
object Node {
val nodeReads: Reads[Node] =
Element.typeRead("node").flatMap(_ => Json.reads[Node])
}
Note that if the constraint on type fails the flatMap call will be bypassed.
The question remains of where to put the method typeRead, in this answer I initially put it in the Element companion object along with the elementReads instance as in the code below.
import play.api.libs.json._
trait Element
object Element {
implicit val elementReads = new Reads[Element] {
override def reads(json: JsValue): JsResult[Element] =
json.validate(
Node.nodeReads.map(_.asInstanceOf[Element]) orElse
Edge.edgeReads.map(_.asInstanceOf[Element])
)
}
def typeRead(`type`: String): Reads[String] = {
val isNotOfType = ValidationError(s"is not of expected type ${`type`}")
(__ \ "type").read[String].filter(isNotOfType)(_ == `type`)
}
}
This is actually a pretty bad place to define typeRead :
- it has nothing specific to Element
- it introduces a circular dependency between the Elementcompanion object and both Node and Edge companion objects
I'll let you think up of the correct location though :)
The specification proving it all works together :
import org.specs2.mutable.Specification
import play.api.libs.json._
import play.api.data.validation.ValidationError
class ElementSpec extends Specification {
"Element reads" should {
"read an edge json as an edge" in {
val result: JsResult[Element] = edgeJson.validate[Element]
result.isSuccess should beTrue
result.get should beEqualTo(Edge("myEdge"))
}
"read a node json as an node" in {
val result: JsResult[Element] = nodeJson.validate[Element]
result.isSuccess should beTrue
result.get should beEqualTo(Node("myNode"))
}
}
"Node reads" should {
"read a node json as an node" in {
val result: JsResult[Node] = nodeJson.validate[Node](Node.nodeReads)
result.isSuccess should beTrue
result.get should beEqualTo(Node("myNode"))
}
"fail to read an edge json as a node" in {
val result: JsResult[Node] = edgeJson.validate[Node](Node.nodeReads)
result.isError should beTrue
val JsError(errors) = result
val invalidNode = JsError.toJson(Seq(
(__ \ "type") -> Seq(ValidationError("is not of expected type node"))
))
JsError.toJson(errors) should beEqualTo(invalidNode)
}
}
"Edge reads" should {
"read a edge json as an edge" in {
val result: JsResult[Edge] = edgeJson.validate[Edge](Edge.edgeReads)
result.isSuccess should beTrue
result.get should beEqualTo(Edge("myEdge"))
}
"fail to read a node json as an edge" in {
val result: JsResult[Edge] = nodeJson.validate[Edge](Edge.edgeReads)
result.isError should beTrue
val JsError(errors) = result
val invalidEdge = JsError.toJson(Seq(
(__ \ "type") -> Seq(ValidationError("is not of expected type edge"))
))
JsError.toJson(errors) should beEqualTo(invalidEdge)
}
}
val edgeJson = Json.parse(
"""
|{
| "type":"edge",
| "name":"myEdge"
|}
""".stripMargin)
val nodeJson = Json.parse(
"""
|{
| "type":"node",
| "name":"myNode"
|}
""".stripMargin)
}
if you don't want to use asInstanceOf as a cast you can write the
elementReads instance like so :
implicit val elementReads = new Reads[Element] {
override def reads(json: JsValue): JsResult[Element] =
json.validate(
Node.nodeReads.map(e => e: Element) orElse
Edge.edgeReads.map(e => e: Element)
)
}
unfortunately, you can't use _ in this case.