Parse a JSON in Scala and create variables for each key - json

I am new to scala and I am trying to parse a JSON shown below
val result = JSON.parseFull("""
{"name": "Naoki", "lang": ["Java", "Scala"] , "positionandvalue": ["5:21", "6:24", "7:6"]}
""")
result: Option[Any] = Some(Map(name -> Naoki, lang -> List(Java, Scala), positionandvalue -> List(5:21, 6:24, 7:6)))
And get the parsed values in a Map
val myMap = result match {
case Some(e) => e
case None => None
}
myMap: Any = Map(name -> Naoki, lang -> List(Java, Scala), positionandvalue -> List(5:21, 6:24, 7:6))
What I need is
1. To get the key as a new variable (to be used as metadata to validate the file) with its corresponding value assigned to it. Something like,
val name = "Naoki"
positionandvalue -> List(5:21, 6:24, 7:6). This variable indicates the List of(Position of string delimited a in file:length of string in position). How can I use this variable to satisfy the requirement.

you cannot dynamically create the variables name and positionandvalue from the Map key. However they can be statically created using the below approach.
val result: Option[Any] = Some(Map("name" -> "Naoki", "lang" -> List("Java", "Scala"), "positionandvalue" -> List("5:21", "6:24", "7:6")))
val myMap: Map[String, Any] = result match {
case Some(e: Map[String, Any] #unchecked) => e
case _ => Map()
}
val name = myMap.get("name") match {
case Some(x: String) => x
case _ => throw new RuntimeException("failure retrieving name key")
}
val positionandvalue = myMap.get("positionandvalue") match {
case Some(x: List[String] #unchecked) => x.map(y => (y.split(":") match {case Array(x1,x2) => x1 -> x2})).toMap
case _ => throw new RuntimeException("failure retrieving positionandvalue key")
}
positionandvalue: scala.collection.immutable.Map[String,String] = Map(5 -> 21, 6 -> 24, 7 -> 6)

Related

Convert Object (can be int, long, string, etc.) to Json Circe

So I have a list of maps with dynamic field and values.
E.g.
val sampleMap = List(
Map("field1" -> 1, "field2" -> "helloworld"),
Map("field3" -> "abcd", "field4" -> 123.34212543))
So basically, I have a variable that is List[Map[String, Object]].
How can I convert this whole thing into a JSON circe?
Nevermind, I found the answer.
Basically we need to match the datatype of the object and convert it to JSON.
def toJson(obj: Object): Json = {
obj match {
case null => Json.Null
case b: java.lang.Boolean => Json.fromBoolean(b)
case i: java.lang.Integer => Json.fromInt(i)
case d: java.lang.Double =>
Json.fromDouble(d).getOrElse(Json.fromDouble(0d).get)
case l: java.lang.Long => Json.fromLong(l)
case t: TemporalAccessor =>
Json.fromString(DtFormatter.format(t))
case u => Json.fromString(String.valueOf(u))
}
}

scala parse json which has List of maps

I have some json when I parsed that it is returning stucture Some(List(Map...))
How to match with something and get the values
Below is the code I tried, I need to get all the map values
import scala.util.parsing.json._
val result = JSON.parseFull("[{\"start\":\"starting\",\"test\":123,\"test2\":324,\"end\":\"ending\"}]")
result match {
case Some(map: Map[String, Any]) => { println(map)
}
case None => println("Parsing failed")
case other => println("Unknown data structure: " + other)
}
but its printing non matching
Unknown data structure: Some(List(Map(start -> starting, test -> 123, test2 -> 324, end -> ending)))
Because of type erasure, you cannot pattern match on generics types. List, Map, Option are generic containers, and in runtime compiler will erase types of these generic containers. e.g. List[String], String will be erased and type will be List[_].
case Some(map: List[Map[String, Any]]) => println(map)
In your case above case, if result is val result: Option[Any] = Some(List(12)) i.e. 12, which type is Int and not Map[String, Any], compiler will still match the result with above case even it expects Map[String, Any]] as type of List.
So, Whats going on?
Its all because of type erasure. The compiler will erase all the types and it won't have any type information at runtime unless you use reflection. That means:
case Some(map: List[Map[String, Any]]) => println(map) is essentially case Some(map: List[_]) => println(map) and therefore, match will success for any type parameter of List e.g. List[Map[String, Any]], List[Map[String, Int]], List[String], List[Int] etc.
Therefore, In case you need to match on such generic container, you have to resolve each container and its nested subtypes explicitly.
def resolve(result: Any): Unit = result match {
case Some(res) if res.isInstanceOf[List[_]] && res.asInstanceOf[List[_]].isEmpty => println("Empty List.") //Some(List())
case Some(res) if res.isInstanceOf[List[_]] && !res.asInstanceOf[List[_]].exists(p => p.isInstanceOf[Map[_, _]] && p.asInstanceOf[Map[_, _]].nonEmpty) => println("List is not empty but each item of List is empty Map.") //Some(List(Map(), Map()))
case Some(res) if res.isInstanceOf[List[_]] && res.asInstanceOf[List[_]].filter(p => p.isInstanceOf[Map[_, _]] && p.asInstanceOf[Map[_, _]].nonEmpty).map(_.asInstanceOf[Map[_,_]]).exists(p => {p.head match {case e if e._1.isInstanceOf[String] && e._2.isInstanceOf[Any] => true; case _ => false}}) => println("Correct data.") // Some(List(Map("key1"-> 1), Map("key2" -> 2)))
case None => println("none")
case other => println("other")
}
val a: Option[Any] = Some(List())
val b: Option[Any] = Some(List(Map(), Map()))
val c: Option[Any] = Some(List(Map("key1"-> 1), Map("key2" -> 2)))
val d: Option[Any] = None
val e: Option[Any] = Some("apple")
resolve(a) // match first case
resolve(b) // match second case
resolve(c) // match third case
resolve(d) // match fourth case
resolve(e) // match fifth case
As has been pointed out the return type is actually Option[List[Map[String, Any]]] so you need to unpick this. However you cannot do this with a single match because of type erasure, so you need to do nested matches to ensure that you have the correct type. This is really tedious, so I thoroughly recommend using something like the Extraction.extract function in json4s that will attempt to match your JSON to a specific Scala type:
type ResultType = List[Map[String, Any]]
def extract(json: JValue)(implicit formats: Formats, mf: Manifest[ResultType]): ResultType =
Extraction.extract[ResultType](json)
If you must do it by hand, it looks something like this:
result match {
case Some(l: List[_]) =>
l.headOption match {
case Some(m) =>
m match {
case m: Map[_,_] =>
m.headOption match {
case Some(p) =>
p match {
case (_: String, _) =>
m.foreach(println(_))
case _ => println("Map key was not String")
}
case _ => println("Map was empty")
}
case _ => println("List did not contain a Map")
}
case _ => println("Result List was empty")
}
case _ => println("Parsing failed")
}
Your output is Option[List[Map[String, Any]]], not Option[Map[String, Any]]. match on the List and you'll be fine:
import scala.util.parsing.json._
val result = JSON.parseFull("[{\"start\":\"starting\",\"test\":123,\"test2\":324,\"end\":\"ending\"}]")
val l: List[Map[String, Any]] = result match {
case Some(list: List[Map[String, Any]]) => list
case _ => throw new Exception("I shouldn't be here") // whatever for a non-match
}
Then you can map (if you want a non-Unit return type)/ foreach (if you don't care about a Unit return type) on that List and do whatever you want it:
l.foreach(println)
l.map(_.toString) // or whatever you want ot do with the Map

How to correctly do Pattern matching by type in this case?

I have a use-case where I'd like to serialize a Saddle Frame to a Scala-Play Json. I had it working for simple element types e.g. Double i.e. Frame[DateTime, String, Double] but now it becomes a bit more hairy trying to serialize Frame types whose elements are complex types e.g. Seq[(DateTime, Double)]
So I have the basic frameWrites implementation that works for primitive element types:
import play.api.libs.json._
import org.saddle._
import org.joda.time._
object JsonUtils {
implicit def frameWrites[RX, CX, E] = new Writes[Frame[RX, CX, E]] {
override def writes(frame: Frame[RX, CX, E]): JsValue = {
val json: JsArray = Json.arr(
(0 until frame.numRows).map { i =>
Json.obj(
frame.rowIx.at(i).toString ->
(0 until frame.numCols).map { j =>
Json.obj(
frame.colIx.at(j).toString -> cellWrites(frame.at(i, j))
)
}
)
})
json
}
}}
So I created a cellWrites to be able to handle complex element types like so:
object JsonUtils {
implicit def cellWrites[E](elem: E)(implicit tag: ClassTag[E]): JsValue = {
elem match {
case seq: Seq[(DateTime, Double)] => {
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this should but doesn't happen
Json.arr(
seq.map { case (dt: DateTime, value: Double) =>
Json.obj(dt.toString -> JsNumber(value))
}
)
}
case doubleValue: Double => JsNumber(doubleValue)
case _ => JsString(elem.toString)
}
}}
The problem is that my Frame or type Frame[String, String, Seq[(DateTime, Double)]] is never matching the appropriate first case in cellWrites. I also tried using the implicit tag type information but could not find the way to use it for Pattern matching. This is a test-case sample Frame I use:
val today = DateTime.parse("2017-11-14T13:00:00.000+01:00")
val colIx = Index("x", "y")
val data = Seq(
Series(
"a" -> Seq(
today.minusHours(2) -> 2.0,
today.minusHours(1) -> 1.0,
today.minusHours(0) -> 0.0
),
"b" -> Seq(
today.minusHours(1) -> 1.0,
today.minusHours(0) -> 0.0
)
),
Series(
"a" -> Seq(
today.minusHours(5) -> 5.0,
today.minusHours(4) -> 4.0,
today.minusHours(3) -> 3.0
),
"b" -> Seq(
today.minusHours(4) -> 4.0,
today.minusHours(3) -> 3.0
)
)
)
val frame: Frame[String, String, Seq[(DateTime, Double)]] = Frame(data, colIx)
println(frame)
Note that all the code above compiles fine. However, the Pattern matching by cell element type is not working.
OK found it, elem was actually of type org.saddle.scalar.Scalar[E] therefore this works:
implicit def cellWrites[E](scalar: Scalar[E]): JsValue = {
val elem = scalar.get
elem match {
case seq: Seq[(DateTime, Double)] => {
Json.arr(
seq.map { case (dt: DateTime, value: Double) =>
Json.obj(dt.toString -> JsNumber(value))
}
)
}
case doubleValue: Double => JsNumber(doubleValue)
case _ => JsString(elem.toString)
}
}

Scala Play Framework JSON JsNull using json4s

I'm new to Scala. How do I handle the JsNull value in my code?
I'm using json4s to convert the JSON to a map.
Should I somehow be converting JsNull to an Option?
Example:
Play JSON : creating json
val jsonA: JsValue = Json.obj(
"name" -> "Bob",
"location" -> "Irvine",
"resident" -> "No",
"nick-name" -> "Bigwig",
"age" -> "6",
"role" -> JsNull,
"car" -> "BMW",
"multiple-residents" -> JsArray(Seq(
JsObject(Seq(
"name" -> JsString("Fiver"),
"age" -> JsNumber(4),
"role" -> JsObject(Seq(
"position" -> JsString("Fiver"),
"" -> JsNumber(4),
"role" -> JsString("janitor")
))
))
))
)
json4s : parsing the json
var jsonAMap:Map[String, Any] = Map()
val jsonAString: String = Json.stringify(jsonA)
jsonAMap = jsonStrToMap(jsonAString)
After the JsValue is converted to a Map using json4s it looks like this:
Map(name -> Bob, location -> Irvine, role -> null, resident -> No, car -> BMW, multiple-residents -> List(Map(name -> Fiver, age -> 4, role -> Map(position -> Fiver, -> 4, role -> janitor))), age -> 6, nick-name -> Bigwig)
When I create a create a List of the values, I end up with a null value in my list. Once I'm pattern matching all the values of the list I end up trying to patter match a null which is not possible (I'm sure I'm not supposed to use a will card for all my cases, but I'm learning) :
for(i <- 0 to beforeValsList.length - 1){
beforeValsList(i) match {
case _ : Map[_,_] =>
compareJson(
beforeValsList(i).asInstanceOf[Map[String,Any]],
afterValsList(i).asInstanceOf[Map[String,Any]],
rdeltaBefore, rdeltaAfter, sameKeyList(i).toString()
)
case _ if (beforeValsList(i) != afterValsList(i)) =>
// if i'm from a recursion, build a new map and add me
// to the deltas as a key->value pair
rdeltaBefore += sameKeyList(i).toString -> beforeValsList(i)
rdeltaAfter += sameKeyList(i).toString -> afterValsList(i)
case _ =>
println("catch all: " + beforeValsList(i).toString
+ " " + afterValsList(i).toString)
}
}
json4s converts JsNull to a null. Should I do a null check:
if(!beforeValsList(i) == null){
beforeValsList(i) match{...}
}
Or is there a way for me to change the null to an Option when I'm putting the values from the Map to a List?
I'm not sure what best practices are and why jsno4s changes JsNull to null instead of an Option, and whether or not that's possible.
Cheers.
I'm still not sure how you would like to handle JsNull, null (or None if you use Option), but you function to get the difference between the Map[String, Any] before and after can be simplified :
type JsonMap = Map[String, Any]
def getMapDiffs(mapBefore: JsonMap, mapAfter: JsonMap) : (JsonMap, JsonMap) = {
val sameKeys = mapBefore.keySet intersect mapAfter.keySet
val startAcc = (Map.empty[String, Any], Map.empty[String, Any])
sameKeys.foldLeft(startAcc){ case (acc # (deltaBefore, deltaAfter), key) =>
(mapBefore(key), mapAfter(key)) match {
// two maps -> add map diff recursively to before diff and after diff
case (beforeMap: Map[_, _], afterMap: Map[_, _]) =>
val (deltaB, deltaA) =
getMapDiffs(beforeMap.asInstanceOf[JsonMap], afterMap.asInstanceOf[JsonMap])
(deltaBefore + (key -> deltaB), deltaAfter + (key -> deltaA))
// values before and after are different
// add values to before diff and after diff
case (beforeValue, afterValue) if beforeValue != afterValue =>
(deltaBefore + (key -> beforeValue), deltaAfter + (key -> afterValue))
// keep existing diff
case _ => acc
}
}
}
Which can be used as:
val (mapBefore, mapAfter) = (
Map("a" -> "alpha", "b" -> "beta", "c" -> "gamma", "d" -> Map("e" -> "epsilon")),
Map("a" -> "alpha", "b" -> List("beta"), "c" -> null, "d" -> Map("e" -> 3))
)
val (deltaBefore, deltaAfter) = getMapDiffs(mapBefore, mapAfter)
// deltaBefore: JsonMap = Map(b -> beta, c -> gamma, d -> Map(e -> epsilon))
// deltaAfter: JsonMap = Map(b -> List(beta), c -> null, d -> Map(e -> 3))
deltaBefore.toList
// List[(String, Any)] = List((b,beta), (c,gamma), (d,Map(e -> epsilon)))

Play Framework: How to replace all the occurrence of a value in a JSON tree

Given the following JSON...
{ "id":"1234",
"name" -> "joe",
"tokens: [{
"id":"1234",
"id":"2345"
}]
}
... I need to replace the value of all the ids by xxxx like this:
{ "id":"xxxx",
"name" -> "joe",
"tokens: [{
"id":"xxxx",
"id":"xxxx"
}]
}
Let's start create the JSON tree:
val json = Json.obj(
"id" -> "1234",
"name" -> "joe",
"tokens" -> Json.arr(
Json.obj("id" -> "1234"),
Json.obj("id" -> "2345")
)
)
json: play.api.libs.json.JsObject = {"id":"1234","name":"joe","tokens":[{"id":"1234"},{"id":"2345"}]}
Then, getting all the ids is very simple:
json \\ "id"
res64: Seq[play.api.libs.json.JsValue] = List("1234", "1234", "2345")
Now, how do I replace the value of all the ids by xxxx?
There doesn't appear to be a nice way to do this with the standard Play JSON library, although I'd be happy to be proved wrong in that regard. You can however do it easily using the play-json-zipper extensions:
import play.api.libs.json._
import play.api.libs.json.extensions._
val json = Json.obj(
"id" -> "1234",
"name" -> "joe",
"tokens" -> Json.arr(
Json.obj("id" -> "1234"),
Json.obj("id" -> "2345")
)
)
// Using `updateAll` we pattern match on a path (ignoring
// the existing value, as long as it's a string) and replace it
val transformed = json.updateAll {
case (__ \ "id", JsString(_)) => JsString("xxxx")
}
// play.api.libs.json.JsValue = {"id":"xxxx","name":"joe","tokens":[{"id":"xxxx"},{"id":"xxxx"}]}
To make that a re-usable function:
def replaceValue(json: JsValue, key: String, replacement: String) = json.updateAll {
case (__ \ path, JsString(_)) if path == key => JsString(replacement)
}
The json-zipper extensions are still "experimental", but if you want to add them to your project add the following to your project/Build.scala appDependencies:
"play-json-zipper" %% "play-json-zipper" % "1.0"
and the following resolver:
"Mandubian repository releases" at "https://github.com/mandubian/mandubian-mvn/raw/master/releases/"
Probably it isn't most efficient way to do it, but you can try to convert your JSON to an object copy it with new fields and then convert it back to json. Unfortunately currently I don't have environment to check the code, but it should be something like this:
case class MyId(id: String)
case class MyObject(id: String, name: String, tokens: List[MyId])
implicit val idFormat = Json.format[MyId]
implicit val objectFormat = Json.format[MyObject]
val json = Json.parse(jsonString)
val jsResult = Json.fromJson[MyObject](json)
val obj = jsResult match {
case JsSuccess(s, _) => s
case _ => throw new IllegalStateException("Unexpected")
}
val newObj = obj.copy(id = "xxxx")
val result = Json.toJson(newObj)