value keys is not a member of play.api.libs.json.JsValue - json

I'm try to get the head from the keys of JsValue type in Scala. I googled a lot to know how to get the head key from JsValue type.
Finally, I found that result.keys.head is the way to get the head key, but it throws error value keys is not a member of play.api.libs.json.JsValue.
And my result variable has the below form of data:
{
"intents": [{
"intent": "feeling",
"confidence": 0.1018563217175903
}],
"entities": [],
"input": {
"text": "{reset-encounter}"
},
"output": "Good"
}
Code:
import play.api.libs.json._
val jsonStr = """
{
"intents": [{
"intent": "feeling",
"confidence": 0.1018563217175903
}],
"entities": [],
"input": {
"text": "{reset-encounter}"
},
"output": "Good"
}
"""
val result = Json.parse(jsonStr)
println("key: ", result.keys.head)
At result.keys.head line, throws error.
I'm not sure but I think, may be I'm doing something wrong here.

Json.parse produces a JsValue, which could represent any type of json object (boolean, number, array, etc). If you know you're working with an object, you can use .as[JsObject]:
import play.api.libs.json._
val result = Json.parse(jsonStr).as[JsObject]
println("key: " + result.keys.head)

What are you trying to get? That's not the way to deal with play.api.Json objects.
.keys would result in a Map, not in a JsValue.
Check the documentation: https://www.playframework.com/documentation/2.5.x/ScalaJson
If you want to access a specific key (https://www.playframework.com/documentation/2.5.x/ScalaJson#Traversing-a-JsValue-structure) you should try:
result \ "keyName"
or for a recursive search:
result \\ "keyName"

Related

How to traverse list of nested maps in scala

I have been given a json string that looks like the following one:
{
"dataflows": [
{
"name": "test",
"sources": [
{
"name": "person_inputs",
"path": "/data/input/events/person/*",
"format": "JSON"
}
],
"transformations": [
{
"name": "validation",
"type": "validate_fields",
"params": {
"input": "person_inputs",
"validations": [
{
"field": "office",
"validations": [
"notEmpty"
]
},
{
"field": "age",
"validations": [
"notNull"
]
}
]
}
},
{
"name": "ok_with_date",
"type": "add_fields",
"params": {
"input": "validation_ok",
"addFields": [
{
"name": "dt",
"function": "current_timestamp"
}
]
}
}
],
"sinks": [
{
"input": "ok_with_date",
"name": "raw-ok",
"paths": [
"/data/output/events/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
},
{
"input": "validation_ko",
"name": "raw-ko",
"paths": [
"/data/output/discards/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
}
And I have been asked to use it as some kind of recipe for an ETL pipeline, i.e., the data must be extracted from the "path" specifid in the "sources" key, the transformations to be carried out are specified within the "transformations" key and, finally, the transformed data must saved to one of the two specified "sink" keys.
I have decided to convert the json string into a scala map, as follows:
val json = Source.fromFile("path/to/json")
//parse
val parsedJson = jsonStrToMap(json.mkString)
implicit val formats = org.json4s.DefaultFormats
val parsedJson = parse(jsonStr).extract[Map[String, Any]]
so, with that, I get a structure like this one:
which is a map whose first value is a list of maps. I can evaluate parsedJson("dataflows") to get:
which is a list, as expected, but, then I cannot traverse such list, even though I need to in order to get to the sources, transformations and sinks. I have tried using the index of the listto, for example, get its first element, like this: parsedJson("dataflows")(0), but to no avail.
Can anyone please help me traverse this structure? Any help would be much appreciated.
Cheers,
When you evaluate parsedJson("dataflows") a Tuple2 is returned aka a Tuple which has two elements that are accessed with ._1 and ._2
So for dataflows(1)._1 the value returned is "sources" and dataflows(1)._2 is list of maps (List[Map[K,V]) which can be traversed like you would normally traverse elements of a List where each element is Map
Let's deconstruct this for example:
val dataFlowsZero = ("sources", List(Map(42 -> "foo"), Map(42 -> "bar")))
The first element in the Tuple
scala> dataFlowsZero._1
String = sources
The second element in the Tuple
scala> dataFlowsZero._2
List[Map[Int, String]] = List(Map(42 -> foo), Map(42 -> bar))`
Map the keys in each Map in List to a new List
scala> dataFlowsZero._2.map(m => m.keys)
List[Iterable[Int]] = List(Set(42), Set(42))
Map the values in each Map in the List to a new List
scala> dataFlowsZero._2.map(m => m.values)
List[Iterable[String]] = List(Iterable(foo), Iterable(bar))
The best solution is to convert the JSON to the full data structure that you have been provided rather than just Map[String, Any]. This makes it trivial to pick out the data that you want. For example,
val dataFlows = parse(jsonStr).extract[DataFlows]
case class DataFlows(dataflows: List[DataFlow])
case class DataFlow(name: String, sources: List[Source], transformations: List[Transformation], sinks: List[Sink])
case class Source(name: String, path: String, format: String)
case class Transformation(name: String, `type`: String, params: List[Param])
case class Param(input: String, validations: List[Validation])
case class Validation(field: String, validations: List[String])
case class Sink(input: String, name: String, paths: List[String], format: String, saveMode: String)
The idea is to make the JSON handler do most of the work to create a type-safe version of the original data.

Scala: Map part of Json to Object with playframework

I'm relatively new to Scala. I would like to map part of my Json to my Object. Code looks like this:
def seasons = (json \\ "season")
case class:
case class Season(startDate: LocalDate, endDate: LocalDate)
json-structure:
[
{
"id": "",
"name": "",
"season": {
"start": "0",
"end": "0"
}
}
]
I would somehow like to end up with a List[Season], so I can loop through it.
Question #2
json-structure:
[
{
"id": "",
"name": "",
"season": {
"start": "0",
"end": "0"
}
},
{
"id": "",
"name": "",
"season": {
"start": "0",
"end": "0"
}
}...
]
Json (which is a JsValue btw) brings multiple regions as can be seen above. Case classed are provided (Region holds a Season), naming is the same as in json.
Formats look like this:
implicit val seasonFormat: Format[Season] = Json.format[Season]
implicit val regionFormat: Format[Region] = Json.format[Region]
So what would I need to call in order to get a List[Region]? I thought of something like regionsJson.as[List[Region]] as I defined the Format, which provides me the Read/Write possibilities. But unfortunately, it's not working.
What is the best way doing this? I've tried it with an JsArray, but I have difficulties with mapping it...
Any input would be much appreciated!
I've added some changes to your original case class and renamed its fields to match json fields.
The following code does parsing of the json into Seq[Session]
import java.time.LocalDate
import play.api.libs.json._
case class Season(start: LocalDate, end: LocalDate)
implicit val sessionFormat: Format[Season] = Json.format[Season]
val json =
"""
|[
| {
| "id": "",
| "name": "",
| "season": {
| "start": "2020-10-20",
| "end": "2020-10-22"
| }
| }
|]
|""".stripMargin
val seasonsJson: collection.Seq[JsValue] = Json.parse(json) \\ "season"
val seasons: collection.Seq[Season] = seasonsJson.map(_.as[Season])
seasons.foreach(println)
Please note that I changed the data of your json and instead of 0, which is not a valid date, I provided dates in iso format yyyy-mm-dd.
The above code works with play-json version 2.9.0.
---UPDATE---
Following up comment by #cchantep.
Method as will produce an exception if json cannot be mapped in case class, a non-exception option is to use asOpt that does not throw an exception but returns a None if mapping is not possible.

Transforming JSON with Circe

Suppose I have some JSON data like this:
{
"data": {
"title": "example input",
"someBoolean": false,
"innerData": {
"innerString": "input inner string",
"innerBoolean": true,
"innerCollection": [1,2,3,4,5]
},
"collection": [6,7,8,9,0]
}
}
And I want to flatten it a bit and transform or remove some fields, to get the following result:
{
"data": {
"ttl": "example input",
"bool": false,
"collection": [6,7,8,9,0],
"innerCollection": [1,2,3,4,5]
}
}
How can I do this with Circe?
(Note that I'm asking this as a FAQ since similar questions often come up in the Circe Gitter channel. This specific example is from a question asked there yesterday.)
I've sometimes said that Circe is primarily a library for encoding and decoding JSON, not for transforming JSON values, and in general I'd recommend mapping to Scala types and then defining relationships between those (as Andriy Plokhotnyuk suggests here), but for many cases writing transformations with cursors works just fine, and in my view this kind of thing is one of them.
Here's how I'd implement this transformation:
import io.circe.{DecodingFailure, Json, JsonObject}
import io.circe.syntax._
def transform(in: Json): Either[DecodingFailure, Json] = {
val someBoolean = in.hcursor.downField("data").downField("someBoolean")
val innerData = someBoolean.delete.downField("innerData")
for {
boolean <- someBoolean.as[Json]
collection <- innerData.get[Json]("innerCollection")
obj <- innerData.delete.up.as[JsonObject]
} yield Json.fromJsonObject(
obj.add("boolean", boolean).add("collection", collection)
)
}
And then:
val Right(json) = io.circe.jawn.parse(
"""{
"data": {
"title": "example input",
"someBoolean": false,
"innerData": {
"innerString": "input inner string",
"innerBoolean": true,
"innerCollection": [1,2,3]
},
"collection": [6,7,8]
}
}"""
)
And:
scala> transform(json)
res1: Either[io.circe.DecodingFailure,io.circe.Json] =
Right({
"data" : {
"title" : "example input",
"collection" : [
6,
7,
8
]
},
"boolean" : false,
"collection" : [
1,
2,
3
]
})
If you look at it the right way, our transform method kind of resembles a decoder, and we can actually write it as one (although I'd definitely recommend not making it implicit):
import io.circe.{Decoder, Json, JsonObject}
import io.circe.syntax._
val transformData: Decoder[Json] = { c =>
val someBoolean = c.downField("data").downField("someBoolean")
val innerData = someBoolean.delete.downField("innerData")
(
innerData.delete.up.as[JsonObject],
someBoolean.as[Json],
innerData.get[Json]("innerCollection")
).mapN(_.add("boolean", _).add("collection", _)).map(Json.fromJsonObject)
}
This can be convenient in some situations where you want to perform the transformation as part of a pipeline that expects a decoder:
scala> io.circe.jawn.decode(myJsonString)(transformData)
res2: Either[io.circe.Error,io.circe.Json] =
Right({
"data" : {
"title" : "example input",
"collection" : [ ...
This is also potentially confusing, though, and I've thought about adding some kind of Transformation type to Circe that would encapsulate transformations like this without questionably repurposing the Decoder type class.
One nice thing about both the transform method and this decoder is that if the input data doesn't have the expected shape, the resulting error will include a history that points to the problem.

How do I ignore decoding failures in a JSON array?

Suppose I want to decode some values from a JSON array into a case class with circe. The following works just fine:
scala> import io.circe.generic.auto._, io.circe.jawn.decode
import io.circe.generic.auto._
import io.circe.jawn.decode
scala> case class Foo(name: String)
defined class Foo
scala> val goodDoc = """[{ "name": "abc" }, { "name": "xyz" }]"""
goodDoc: String = [{ "name": "abc" }, { "name": "xyz" }]
scala> decode[List[Foo]](goodDoc)
res0: Either[io.circe.Error,List[Foo]] = Right(List(Foo(abc), Foo(xyz)))
It's sometimes the case that the JSON array I'm decoding contains other, non-Foo-shaped stuff, though, which results in a decoding error:
scala> val badDoc =
| """[{ "name": "abc" }, { "id": 1 }, true, "garbage", { "name": "xyz" }]"""
badDoc: String = [{ "name": "abc" }, { "id": 1 }, true, "garbage", { "name": "xyz" }]
scala> decode[List[Foo]](badDoc)
res1: Either[io.circe.Error,List[Foo]] = Left(DecodingFailure(Attempt to decode value on failed cursor, List(DownField(name), MoveRight, DownArray)))
How can I write a decoder that ignores anything in the array that can't be decoded into my case class?
The most straightforward way to solve this problem is to use a decoder that first tries to decode each value as a Foo, and then falls back to the identity decoder if the Foo decoder fails. The new either method in circe 0.9 makes the generic version of this practically a one-liner:
import io.circe.{ Decoder, Json }
def decodeListTolerantly[A: Decoder]: Decoder[List[A]] =
Decoder.decodeList(Decoder[A].either(Decoder[Json])).map(
_.flatMap(_.left.toOption)
)
It works like this:
scala> val myTolerantFooDecoder = decodeListTolerantly[Foo]
myTolerantFooDecoder: io.circe.Decoder[List[Foo]] = io.circe.Decoder$$anon$21#2b48626b
scala> decode(badDoc)(myTolerantFooDecoder)
res2: Either[io.circe.Error,List[Foo]] = Right(List(Foo(abc), Foo(xyz)))
To break down the steps:
Decoder.decodeList says "define a list decoder that tries to use the given decoder to decode each JSON array value".
Decoder[A].either(Decoder[Json] says "first try to decode the value as an A, and if that fails decode it as a Json value (which will always succeed), and return the result (if any) as a Either[A, Json]".
.map(_.flatMap(_.left.toOption)) says "take the resulting list of Either[A, Json] values and remove all the Rights".
…which does what we want in a fairly concise, compositional way. At some point we might want to bundle this up into a utility method in circe itself, but for now writing out this explicit version isn't too bad.

extract case classes from json file scala play

im trying to extract my data from json into a case class without success.
the Json file:
[
{
"name": "bb",
"loc": "sss",
"elements": [
{
"name": "name1",
"loc": "firstHere",
"elements": []
}
]
},
{
"name": "ca",
"loc": "sss",
"elements": []
}
]
my code :
case class ElementContainer(name : String, location : String,elements : Seq[ElementContainer])
object elementsFormatter {
implicit val elementFormatter = Json.format[ElementContainer]
}
object Applicationss extends App {
val el = new ElementContainer("name1", "firstHere", Seq.empty)
val el1Cont = new ElementContainer("bb","sss", Seq(el))
val source:String=Source.fromFile("src/bin/elementsTree.json").getLines.mkString
val jsonFormat = Json.parse(source)
val r1= Json.fromJson[ElementContainer](jsonFormat)
}
after running this im getting inside r1:
JsError(List((/elements,List(ValidationError(List(error.path.missing),WrappedArray()))), (/name,List(ValidationError(List(error.path.missing),WrappedArray()))), (/location,List(ValidationError(List(error.path.missing),WrappedArray())))))
been trying to extract this data forever, please advise
You have location instead loc and, you'll need to parse file into a Seq[ElementContainer], since it's an array, not a single ElementContainer:
Json.fromJson[Seq[ElementContainer]](jsonFormat)
Also, you have the validate method that will return you either errors or parsed json object..