Scala Play Framework JSON JsNull using json4s - json

I'm new to Scala. How do I handle the JsNull value in my code?
I'm using json4s to convert the JSON to a map.
Should I somehow be converting JsNull to an Option?
Example:
Play JSON : creating json
val jsonA: JsValue = Json.obj(
"name" -> "Bob",
"location" -> "Irvine",
"resident" -> "No",
"nick-name" -> "Bigwig",
"age" -> "6",
"role" -> JsNull,
"car" -> "BMW",
"multiple-residents" -> JsArray(Seq(
JsObject(Seq(
"name" -> JsString("Fiver"),
"age" -> JsNumber(4),
"role" -> JsObject(Seq(
"position" -> JsString("Fiver"),
"" -> JsNumber(4),
"role" -> JsString("janitor")
))
))
))
)
json4s : parsing the json
var jsonAMap:Map[String, Any] = Map()
val jsonAString: String = Json.stringify(jsonA)
jsonAMap = jsonStrToMap(jsonAString)
After the JsValue is converted to a Map using json4s it looks like this:
Map(name -> Bob, location -> Irvine, role -> null, resident -> No, car -> BMW, multiple-residents -> List(Map(name -> Fiver, age -> 4, role -> Map(position -> Fiver, -> 4, role -> janitor))), age -> 6, nick-name -> Bigwig)
When I create a create a List of the values, I end up with a null value in my list. Once I'm pattern matching all the values of the list I end up trying to patter match a null which is not possible (I'm sure I'm not supposed to use a will card for all my cases, but I'm learning) :
for(i <- 0 to beforeValsList.length - 1){
beforeValsList(i) match {
case _ : Map[_,_] =>
compareJson(
beforeValsList(i).asInstanceOf[Map[String,Any]],
afterValsList(i).asInstanceOf[Map[String,Any]],
rdeltaBefore, rdeltaAfter, sameKeyList(i).toString()
)
case _ if (beforeValsList(i) != afterValsList(i)) =>
// if i'm from a recursion, build a new map and add me
// to the deltas as a key->value pair
rdeltaBefore += sameKeyList(i).toString -> beforeValsList(i)
rdeltaAfter += sameKeyList(i).toString -> afterValsList(i)
case _ =>
println("catch all: " + beforeValsList(i).toString
+ " " + afterValsList(i).toString)
}
}
json4s converts JsNull to a null. Should I do a null check:
if(!beforeValsList(i) == null){
beforeValsList(i) match{...}
}
Or is there a way for me to change the null to an Option when I'm putting the values from the Map to a List?
I'm not sure what best practices are and why jsno4s changes JsNull to null instead of an Option, and whether or not that's possible.
Cheers.

I'm still not sure how you would like to handle JsNull, null (or None if you use Option), but you function to get the difference between the Map[String, Any] before and after can be simplified :
type JsonMap = Map[String, Any]
def getMapDiffs(mapBefore: JsonMap, mapAfter: JsonMap) : (JsonMap, JsonMap) = {
val sameKeys = mapBefore.keySet intersect mapAfter.keySet
val startAcc = (Map.empty[String, Any], Map.empty[String, Any])
sameKeys.foldLeft(startAcc){ case (acc # (deltaBefore, deltaAfter), key) =>
(mapBefore(key), mapAfter(key)) match {
// two maps -> add map diff recursively to before diff and after diff
case (beforeMap: Map[_, _], afterMap: Map[_, _]) =>
val (deltaB, deltaA) =
getMapDiffs(beforeMap.asInstanceOf[JsonMap], afterMap.asInstanceOf[JsonMap])
(deltaBefore + (key -> deltaB), deltaAfter + (key -> deltaA))
// values before and after are different
// add values to before diff and after diff
case (beforeValue, afterValue) if beforeValue != afterValue =>
(deltaBefore + (key -> beforeValue), deltaAfter + (key -> afterValue))
// keep existing diff
case _ => acc
}
}
}
Which can be used as:
val (mapBefore, mapAfter) = (
Map("a" -> "alpha", "b" -> "beta", "c" -> "gamma", "d" -> Map("e" -> "epsilon")),
Map("a" -> "alpha", "b" -> List("beta"), "c" -> null, "d" -> Map("e" -> 3))
)
val (deltaBefore, deltaAfter) = getMapDiffs(mapBefore, mapAfter)
// deltaBefore: JsonMap = Map(b -> beta, c -> gamma, d -> Map(e -> epsilon))
// deltaAfter: JsonMap = Map(b -> List(beta), c -> null, d -> Map(e -> 3))
deltaBefore.toList
// List[(String, Any)] = List((b,beta), (c,gamma), (d,Map(e -> epsilon)))

Related

Modify json values from list of maps and save output

I have the following example json that I read in using the play framework.
{
"field_a": "dummy",
"field_b": "dummy",
"nest": {
"nest_a": "dummy",
"nest_b": 87
},
"field_c": null,
"field_d": null,
"field_e": "chocolate",
"field_f": "sugar",
"array": [
"dummy entry"
],
"id": "Anything"
}
I Then have the following List of Maps that I want to swap out data with which is my Input:
val substitutionsList: List[mutable.Map[String, String]] = List(
mutable.Map("field_b" -> "dummy string", "field_d" -> "2016-01-01", "field_f" -> "2011-01-01"),
mutable.Map("field_b" -> "dummy string", "field_d" -> "2018-01-01", "field_f" -> "2018-01-01"),
mutable.Map("field_b" -> "dummy string", "field_d" -> "2018-04-01", "field_f" -> "2018-04-01"),
mutable.Map("field_b" -> "dummy string", "field_d" -> "2016-01-01", "field_f" -> "2016-01-01")
)
I am reading in the json as follows:
def parseSchemaJson(schemaContent: String) = Json.parse(schemaContent).as[JsObject]
val baseSchemaInput = parseSchemaJson(Source.fromFile("/dummy.json").mkString)
I want to iterate over my Input and swap out the values in the json for the values in my map and after each one is done, create a new .json file.
private def replaceField(json: JsObject, fieldToReplace: String): Option[String] = (json \ fieldToReplace).asOpt[String]
println(replaceField(baseSchemaInput, "field_a")) //prints dummy
I can list out the value in my json using something like this but I have no idea how to swap the value from my list into each respective bit and write out a json file.
First occurance of the expected output
{
"field_a": "dummy",
"field_b": "dummy string",
"nest": {
"nest_a": "dummy",
"nest_b": 87
},
"field_c": null,
"field_d": "2016-01-01",
"field_e": "chocolate",
"field_f": "2011-01-01",
"array": [
"dummy entry"
],
"id": "Anything"
}
Given the following substitution list:
val substitutionsList: List[mutable.Map[String, Any]] = List(
mutable.Map("field_b" -> "dummy string 1", "field_d" -> "2016-01-01", "field_f" -> "2011-01-01"),
mutable.Map("field_b" -> "dummy string 2", "nest" -> mutable.Map("nest_b" -> "90"))
)
Note that the second one updates a nested value. You need to create a nested Map to define nested values.
You can define a function to transform a Map[String, Any] (Any because we can have either a String or a Map as value) into a JsObject. This is a recursive function and will call itself in case the Value is a Map
def mapToJsObject(map: mutable.Map[String, Any]): JsObject =
JsObject(map.mapValues {
case v:String => JsString(v)
case v:mutable.Map[String, mutable.Map[String, Any]] => mapToJsObject(v.asInstanceOf[mutable.Map[String, Any]])
})
Then go through your list of substitutions and deep merge the JsObject made from each Map with the original JsObject using the deepMerge function defined on the JsObject class. It will merge the nested objects too. See API here
val substitutedJsObjects: List[JsObject] = substitutionsList
.map(mapToJsObject)
.map(baseSchemaInput.deepMerge)
This should give you a list of JsObjects, one per Map in your List
You can then write them to file. Here is an example to write one file per json string. Files will be named 0.json, 1.json, etc ..
def writeToFile(jsObject: JsObject, fileName: String): Unit = {
println("writing "+fileName)
val pw = new PrintWriter(new File(fileName))
pw.write(jsObject.toString())
pw.close()
}
substitutedJsObjects.zipWithIndex.foreach {
case (jsObject, index) => {
val fileName = index.toString + ".json"
writeToFile(jsObject, fileName)
}
}

How to correctly do Pattern matching by type in this case?

I have a use-case where I'd like to serialize a Saddle Frame to a Scala-Play Json. I had it working for simple element types e.g. Double i.e. Frame[DateTime, String, Double] but now it becomes a bit more hairy trying to serialize Frame types whose elements are complex types e.g. Seq[(DateTime, Double)]
So I have the basic frameWrites implementation that works for primitive element types:
import play.api.libs.json._
import org.saddle._
import org.joda.time._
object JsonUtils {
implicit def frameWrites[RX, CX, E] = new Writes[Frame[RX, CX, E]] {
override def writes(frame: Frame[RX, CX, E]): JsValue = {
val json: JsArray = Json.arr(
(0 until frame.numRows).map { i =>
Json.obj(
frame.rowIx.at(i).toString ->
(0 until frame.numCols).map { j =>
Json.obj(
frame.colIx.at(j).toString -> cellWrites(frame.at(i, j))
)
}
)
})
json
}
}}
So I created a cellWrites to be able to handle complex element types like so:
object JsonUtils {
implicit def cellWrites[E](elem: E)(implicit tag: ClassTag[E]): JsValue = {
elem match {
case seq: Seq[(DateTime, Double)] => {
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this should but doesn't happen
Json.arr(
seq.map { case (dt: DateTime, value: Double) =>
Json.obj(dt.toString -> JsNumber(value))
}
)
}
case doubleValue: Double => JsNumber(doubleValue)
case _ => JsString(elem.toString)
}
}}
The problem is that my Frame or type Frame[String, String, Seq[(DateTime, Double)]] is never matching the appropriate first case in cellWrites. I also tried using the implicit tag type information but could not find the way to use it for Pattern matching. This is a test-case sample Frame I use:
val today = DateTime.parse("2017-11-14T13:00:00.000+01:00")
val colIx = Index("x", "y")
val data = Seq(
Series(
"a" -> Seq(
today.minusHours(2) -> 2.0,
today.minusHours(1) -> 1.0,
today.minusHours(0) -> 0.0
),
"b" -> Seq(
today.minusHours(1) -> 1.0,
today.minusHours(0) -> 0.0
)
),
Series(
"a" -> Seq(
today.minusHours(5) -> 5.0,
today.minusHours(4) -> 4.0,
today.minusHours(3) -> 3.0
),
"b" -> Seq(
today.minusHours(4) -> 4.0,
today.minusHours(3) -> 3.0
)
)
)
val frame: Frame[String, String, Seq[(DateTime, Double)]] = Frame(data, colIx)
println(frame)
Note that all the code above compiles fine. However, the Pattern matching by cell element type is not working.
OK found it, elem was actually of type org.saddle.scalar.Scalar[E] therefore this works:
implicit def cellWrites[E](scalar: Scalar[E]): JsValue = {
val elem = scalar.get
elem match {
case seq: Seq[(DateTime, Double)] => {
Json.arr(
seq.map { case (dt: DateTime, value: Double) =>
Json.obj(dt.toString -> JsNumber(value))
}
)
}
case doubleValue: Double => JsNumber(doubleValue)
case _ => JsString(elem.toString)
}
}

Parse a JSON in Scala and create variables for each key

I am new to scala and I am trying to parse a JSON shown below
val result = JSON.parseFull("""
{"name": "Naoki", "lang": ["Java", "Scala"] , "positionandvalue": ["5:21", "6:24", "7:6"]}
""")
result: Option[Any] = Some(Map(name -> Naoki, lang -> List(Java, Scala), positionandvalue -> List(5:21, 6:24, 7:6)))
And get the parsed values in a Map
val myMap = result match {
case Some(e) => e
case None => None
}
myMap: Any = Map(name -> Naoki, lang -> List(Java, Scala), positionandvalue -> List(5:21, 6:24, 7:6))
What I need is
1. To get the key as a new variable (to be used as metadata to validate the file) with its corresponding value assigned to it. Something like,
val name = "Naoki"
positionandvalue -> List(5:21, 6:24, 7:6). This variable indicates the List of(Position of string delimited a in file:length of string in position). How can I use this variable to satisfy the requirement.
you cannot dynamically create the variables name and positionandvalue from the Map key. However they can be statically created using the below approach.
val result: Option[Any] = Some(Map("name" -> "Naoki", "lang" -> List("Java", "Scala"), "positionandvalue" -> List("5:21", "6:24", "7:6")))
val myMap: Map[String, Any] = result match {
case Some(e: Map[String, Any] #unchecked) => e
case _ => Map()
}
val name = myMap.get("name") match {
case Some(x: String) => x
case _ => throw new RuntimeException("failure retrieving name key")
}
val positionandvalue = myMap.get("positionandvalue") match {
case Some(x: List[String] #unchecked) => x.map(y => (y.split(":") match {case Array(x1,x2) => x1 -> x2})).toMap
case _ => throw new RuntimeException("failure retrieving positionandvalue key")
}
positionandvalue: scala.collection.immutable.Map[String,String] = Map(5 -> 21, 6 -> 24, 7 -> 6)

Play Framework: How to replace all the occurrence of a value in a JSON tree

Given the following JSON...
{ "id":"1234",
"name" -> "joe",
"tokens: [{
"id":"1234",
"id":"2345"
}]
}
... I need to replace the value of all the ids by xxxx like this:
{ "id":"xxxx",
"name" -> "joe",
"tokens: [{
"id":"xxxx",
"id":"xxxx"
}]
}
Let's start create the JSON tree:
val json = Json.obj(
"id" -> "1234",
"name" -> "joe",
"tokens" -> Json.arr(
Json.obj("id" -> "1234"),
Json.obj("id" -> "2345")
)
)
json: play.api.libs.json.JsObject = {"id":"1234","name":"joe","tokens":[{"id":"1234"},{"id":"2345"}]}
Then, getting all the ids is very simple:
json \\ "id"
res64: Seq[play.api.libs.json.JsValue] = List("1234", "1234", "2345")
Now, how do I replace the value of all the ids by xxxx?
There doesn't appear to be a nice way to do this with the standard Play JSON library, although I'd be happy to be proved wrong in that regard. You can however do it easily using the play-json-zipper extensions:
import play.api.libs.json._
import play.api.libs.json.extensions._
val json = Json.obj(
"id" -> "1234",
"name" -> "joe",
"tokens" -> Json.arr(
Json.obj("id" -> "1234"),
Json.obj("id" -> "2345")
)
)
// Using `updateAll` we pattern match on a path (ignoring
// the existing value, as long as it's a string) and replace it
val transformed = json.updateAll {
case (__ \ "id", JsString(_)) => JsString("xxxx")
}
// play.api.libs.json.JsValue = {"id":"xxxx","name":"joe","tokens":[{"id":"xxxx"},{"id":"xxxx"}]}
To make that a re-usable function:
def replaceValue(json: JsValue, key: String, replacement: String) = json.updateAll {
case (__ \ path, JsString(_)) if path == key => JsString(replacement)
}
The json-zipper extensions are still "experimental", but if you want to add them to your project add the following to your project/Build.scala appDependencies:
"play-json-zipper" %% "play-json-zipper" % "1.0"
and the following resolver:
"Mandubian repository releases" at "https://github.com/mandubian/mandubian-mvn/raw/master/releases/"
Probably it isn't most efficient way to do it, but you can try to convert your JSON to an object copy it with new fields and then convert it back to json. Unfortunately currently I don't have environment to check the code, but it should be something like this:
case class MyId(id: String)
case class MyObject(id: String, name: String, tokens: List[MyId])
implicit val idFormat = Json.format[MyId]
implicit val objectFormat = Json.format[MyObject]
val json = Json.parse(jsonString)
val jsResult = Json.fromJson[MyObject](json)
val obj = jsResult match {
case JsSuccess(s, _) => s
case _ => throw new IllegalStateException("Unexpected")
}
val newObj = obj.copy(id = "xxxx")
val result = Json.toJson(newObj)

Scala to JSON in Play Framework 2.1

I'm trying to convert Scala to JSON in the 2.1RC Play Framework.
I can do the following and get JSON:
import play.api.libs.json._
val a1=Map("val1"->"a", "val2"->"b")
Json.toJSon(a1)
Because a1 is just Map[String,String] that works OK.
But if I have something more complex like where I have Map[String,Object], that doesn't work:
val a = Map("val1" -> "xxx", "val2"-> List("a", "b", "c"))
Json.toJSon(a1)
>>> error: No Json deserializer found for type scala.collection.immutable.Map[String,Object]
I found that I can do something like the following:
val a2 = Map("val1" -> Json.toJson("a"), "val2" -> Json.toJson(List("a", "b", "c")))
Json.toJson(a2)
And that works.
But how can I do that in a general way? I thought that I could do something like the following:
a.map{ case(k,v)=> (k, Json.toJson(v) )}
>>> error: No Json deserializer found for type Object
But I still get an error that it can't be deserialized
Additional Information:
Json.toJson can convert a Map[String, String] to a JsValue:
scala> val b = Map( "1" -> "A", "2" -> "B", "3" -> "C", "4" -> "D" )
b: scala.collection.immutable.Map[String,String] = Map(1 -> A, 2 -> B, 3 -> C, 4 -> D)
scala> Json.toJson(b)
res31: play.api.libs.json.JsValue = {"1":"A","2":"B","3":"C","4":"D"}
But, it fails in trying to convert a Map[String, Object]:
scala> a
res34: scala.collection.immutable.Map[String,Object] = Map(val1 -> xxx, val2 -> List(a, b, c))
scala> Json.toJson(a)
<console>:12: error: No Json deserializer found for type scala.collection.immutable.Map[String,Object]. Try to implement an implicit Writes or Format for this type.
Json.toJson(a)
Using the 'hint' from this Play Framework page on converting Scala to Json, I found the following (http://www.playframework.org/documentation/2.0.1/ScalaJson):
If instead of Map[String, Object], there is a Map[String, JsValue], then Json.toJson() will work:
scala> val c = Map("aa" -> Json.toJson("xxxx"), "bb" -> Json.toJson( List("11", "22", "33") ) )
c: scala.collection.immutable.Map[String,play.api.libs.json.JsValue] = Map(aa -> "xxxx", bb -> ["11","22","33"])
scala> Json.toJson(c)
res36: play.api.libs.json.JsValue = {"aa":"xxxx","bb":["11","22","33"]}
So, what I would like, is that given a Map[String, Object], where I know that the Object values were all originally of type String or List[String], how to apply the function Json.toJson() to the all the values in the map and get a Map[String, JsValue].
I also found that I can filter out those values that are purely string and those that are (were) of type List[String]:
scala> val a1 = a.filter({case(k,v) => v.isInstanceOf[String]})
a1: scala.collection.immutable.Map[String,Object] = Map(val1 -> xxx)
scala> val a2 = a.filter({case(k,v) => v.isInstanceOf[List[String]]})
<console>:11: warning: non-variable type argument String in type List[String] is unchecked since it is eliminated by erasure
val a2 = a.filter({case(k,v) => v.isInstanceOf[List[String]]})
^
a2: scala.collection.immutable.Map[String,Object] = Map(val2 -> List(a, b, c))
The List[String] filtering gives a warning, but seems to give the answer I want. If the two filters could be applied and then Json.toJson() used on the values of the result, and the results combined, maybe that would work?
But the filtered results are still of type Map[String, Object] which causes a problem:
scala> Json.toJson(a1)
<console>:13: error: No Json deserializer found for type scala.collection.immutable.Map[String,Object]. Try to implement an implicit Writes or Format for this type.
Json.toJson(a1)
Play 2.1 JSON API does not provide a serializer for the Type Map[String, Ojbect].
Define case class and Format for the specific type instead of Map[String, Object]:
// { "val1" : "xxx", "val2" : ["a", "b", "c"] }
case class Hoge(val1: String, val2: List[String])
implicit val hogeFormat = Json.format[Hoge]
If you don't want to create case class.
The following code provides JSON serializer/deserializer for Map[String, Object]:
implicit val objectMapFormat = new Format[Map[String, Object]] {
def writes(map: Map[String, Object]): JsValue =
Json.obj(
"val1" -> map("val1").asInstanceOf[String],
"val2" -> map("val2").asInstanceOf[List[String]]
)
def reads(jv: JsValue): JsResult[Map[String, Object]] =
JsSuccess(Map("val1" -> (jv \ "val1").as[String], "val2" -> (jv \ "val2").as[List[String]]))
}
More dynamically
import play.api.libs.json._
import play.api.libs.json.Reads._
import play.api.libs.json.Json.JsValueWrapper
implicit val objectMapFormat = new Format[Map[String, Object]] {
def writes(map: Map[String, Object]): JsValue =
Json.obj(map.map{case (s, o) =>
val ret:(String, JsValueWrapper) = o match {
case _:String => s -> JsString(o.asInstanceOf[String])
case _ => s -> JsArray(o.asInstanceOf[List[String]].map(JsString(_)))
}
ret
}.toSeq:_*)
def reads(jv: JsValue): JsResult[Map[String, Object]] =
JsSuccess(jv.as[Map[String, JsValue]].map{case (k, v) =>
k -> (v match {
case s:JsString => s.as[String]
case l => l.as[List[String]]
})
})
}
Sample code:
val jv = Json.toJson(Map("val1" -> "xxx", "val2" -> List("a", "b", "c"), "val3" -> "sss", "val4" -> List("d", "e", "f")))
println(jv)
val jr = Json.fromJson[Map[String, Object]](jv)
println(jr.get)
The output:
> {"val1":"xxx","val2":["a","b","c"],"val3":"sss","val4":["d","e","f"]}
> Map(val1 -> xxx, val2 -> List(a, b, c), val3 -> sss, val4 -> List(d, e, f))