How to insert an empty object into JSON using Circe? - json

I'm getting a JSON object over the network, as a String. I'm then using Circe to parse it. I want to add a handful of fields to it, and then pass it on downstream.
Almost all of that works.
The problem is that my "adding" is really "overwriting". That's actually ok, as long as I add an empty object first. How can I add such an empty object?
So looking at the code below, I am overwriting "sometimes_empty:{}" and it works. But because sometimes_empty is not always empty, it results in some data loss. I'd like to add a field like: "custom:{}" and then ovewrite the value of custom with my existing code.
Two StackOverflow posts were helpful. One worked, but wasn't quite what I was looking for. The other I couldn't get to work.
1: Modifying a JSON array in Scala with circe
2: Adding field to a json using Circe
val js: String = """
{
"id": "19",
"type": "Party",
"field": {
"id": 1482,
"name": "Anne Party",
"url": "https"
},
"sometimes_empty": {
},
"bool": true,
"timestamp": "2018-12-18T11:39:18Z"
}
"""
val newJson = parse(js).toOption
.flatMap { doc =>
doc.hcursor
.downField("sometimes_empty")
.withFocus(_ =>
Json.fromFields(
Seq(
("myUrl", Json.fromString(myUrl)),
("valueZ", Json.fromString(valueZ)),
("valueQ", Json.fromString(valueQ)),
("balloons", Json.fromString(balloons))
)
)
)
.top
}
newJson match {
case Some(v) => return v.toString
case None => println("Failure!")
}

We need to do a couple of things. First, we need to zoom in on the specific property we want to update, if it doesn't exist, we'll create a new empty one. Then, we turn the zoomed in property in the form of a Json into JsonObject in order to be able to modify it using the +: method. Once we've done that, we need to take the updated property and re-introduce it in the original parsed JSON to get the complete result:
import io.circe.{Json, JsonObject, parser}
import io.circe.syntax._
object JsonTest {
def main(args: Array[String]): Unit = {
val js: String =
"""
|{
| "id": "19",
| "type": "Party",
| "field": {
| "id": 1482,
| "name": "Anne Party",
| "url": "https"
| },
| "bool": true,
| "timestamp": "2018-12-18T11:39:18Z"
|}
""".stripMargin
val maybeAppendedJson =
for {
json <- parser.parse(js).toOption
sometimesEmpty <- json.hcursor
.downField("sometimes_empty")
.focus
.orElse(Option(Json.fromJsonObject(JsonObject.empty)))
jsonObject <- json.asObject
emptyFieldJson <- sometimesEmpty.asObject
appendedField = emptyFieldJson.+:("added", Json.fromBoolean(true))
res = jsonObject.+:("sometimes_empty", appendedField.asJson)
} yield res
maybeAppendedJson.foreach(obj => println(obj.asJson.spaces2))
}
}
Yields:
{
"id" : "19",
"type" : "Party",
"field" : {
"id" : 1482,
"name" : "Anne Party",
"url" : "https"
},
"sometimes_empty" : {
"added" : true,
"someProperty" : true
},
"bool" : true,
"timestamp" : "2018-12-18T11:39:18Z"
}

Related

How to traverse list of nested maps in scala

I have been given a json string that looks like the following one:
{
"dataflows": [
{
"name": "test",
"sources": [
{
"name": "person_inputs",
"path": "/data/input/events/person/*",
"format": "JSON"
}
],
"transformations": [
{
"name": "validation",
"type": "validate_fields",
"params": {
"input": "person_inputs",
"validations": [
{
"field": "office",
"validations": [
"notEmpty"
]
},
{
"field": "age",
"validations": [
"notNull"
]
}
]
}
},
{
"name": "ok_with_date",
"type": "add_fields",
"params": {
"input": "validation_ok",
"addFields": [
{
"name": "dt",
"function": "current_timestamp"
}
]
}
}
],
"sinks": [
{
"input": "ok_with_date",
"name": "raw-ok",
"paths": [
"/data/output/events/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
},
{
"input": "validation_ko",
"name": "raw-ko",
"paths": [
"/data/output/discards/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
}
And I have been asked to use it as some kind of recipe for an ETL pipeline, i.e., the data must be extracted from the "path" specifid in the "sources" key, the transformations to be carried out are specified within the "transformations" key and, finally, the transformed data must saved to one of the two specified "sink" keys.
I have decided to convert the json string into a scala map, as follows:
val json = Source.fromFile("path/to/json")
//parse
val parsedJson = jsonStrToMap(json.mkString)
implicit val formats = org.json4s.DefaultFormats
val parsedJson = parse(jsonStr).extract[Map[String, Any]]
so, with that, I get a structure like this one:
which is a map whose first value is a list of maps. I can evaluate parsedJson("dataflows") to get:
which is a list, as expected, but, then I cannot traverse such list, even though I need to in order to get to the sources, transformations and sinks. I have tried using the index of the listto, for example, get its first element, like this: parsedJson("dataflows")(0), but to no avail.
Can anyone please help me traverse this structure? Any help would be much appreciated.
Cheers,
When you evaluate parsedJson("dataflows") a Tuple2 is returned aka a Tuple which has two elements that are accessed with ._1 and ._2
So for dataflows(1)._1 the value returned is "sources" and dataflows(1)._2 is list of maps (List[Map[K,V]) which can be traversed like you would normally traverse elements of a List where each element is Map
Let's deconstruct this for example:
val dataFlowsZero = ("sources", List(Map(42 -> "foo"), Map(42 -> "bar")))
The first element in the Tuple
scala> dataFlowsZero._1
String = sources
The second element in the Tuple
scala> dataFlowsZero._2
List[Map[Int, String]] = List(Map(42 -> foo), Map(42 -> bar))`
Map the keys in each Map in List to a new List
scala> dataFlowsZero._2.map(m => m.keys)
List[Iterable[Int]] = List(Set(42), Set(42))
Map the values in each Map in the List to a new List
scala> dataFlowsZero._2.map(m => m.values)
List[Iterable[String]] = List(Iterable(foo), Iterable(bar))
The best solution is to convert the JSON to the full data structure that you have been provided rather than just Map[String, Any]. This makes it trivial to pick out the data that you want. For example,
val dataFlows = parse(jsonStr).extract[DataFlows]
case class DataFlows(dataflows: List[DataFlow])
case class DataFlow(name: String, sources: List[Source], transformations: List[Transformation], sinks: List[Sink])
case class Source(name: String, path: String, format: String)
case class Transformation(name: String, `type`: String, params: List[Param])
case class Param(input: String, validations: List[Validation])
case class Validation(field: String, validations: List[String])
case class Sink(input: String, name: String, paths: List[String], format: String, saveMode: String)
The idea is to make the JSON handler do most of the work to create a type-safe version of the original data.

Scala: Map part of Json to Object with playframework

I'm relatively new to Scala. I would like to map part of my Json to my Object. Code looks like this:
def seasons = (json \\ "season")
case class:
case class Season(startDate: LocalDate, endDate: LocalDate)
json-structure:
[
{
"id": "",
"name": "",
"season": {
"start": "0",
"end": "0"
}
}
]
I would somehow like to end up with a List[Season], so I can loop through it.
Question #2
json-structure:
[
{
"id": "",
"name": "",
"season": {
"start": "0",
"end": "0"
}
},
{
"id": "",
"name": "",
"season": {
"start": "0",
"end": "0"
}
}...
]
Json (which is a JsValue btw) brings multiple regions as can be seen above. Case classed are provided (Region holds a Season), naming is the same as in json.
Formats look like this:
implicit val seasonFormat: Format[Season] = Json.format[Season]
implicit val regionFormat: Format[Region] = Json.format[Region]
So what would I need to call in order to get a List[Region]? I thought of something like regionsJson.as[List[Region]] as I defined the Format, which provides me the Read/Write possibilities. But unfortunately, it's not working.
What is the best way doing this? I've tried it with an JsArray, but I have difficulties with mapping it...
Any input would be much appreciated!
I've added some changes to your original case class and renamed its fields to match json fields.
The following code does parsing of the json into Seq[Session]
import java.time.LocalDate
import play.api.libs.json._
case class Season(start: LocalDate, end: LocalDate)
implicit val sessionFormat: Format[Season] = Json.format[Season]
val json =
"""
|[
| {
| "id": "",
| "name": "",
| "season": {
| "start": "2020-10-20",
| "end": "2020-10-22"
| }
| }
|]
|""".stripMargin
val seasonsJson: collection.Seq[JsValue] = Json.parse(json) \\ "season"
val seasons: collection.Seq[Season] = seasonsJson.map(_.as[Season])
seasons.foreach(println)
Please note that I changed the data of your json and instead of 0, which is not a valid date, I provided dates in iso format yyyy-mm-dd.
The above code works with play-json version 2.9.0.
---UPDATE---
Following up comment by #cchantep.
Method as will produce an exception if json cannot be mapped in case class, a non-exception option is to use asOpt that does not throw an exception but returns a None if mapping is not possible.

Select a particular value from a JSON array

I have a JSON array containing the following details, I would like to extract the text Alignment value of Right and assign it to a val.
"data":[
{
"formatType": "text",
"value": "bgyufcie huis huids hufhsduhfsl hd"
},
{
"formatType": "text size",
"value": 12
},
{
"formatType": "text alignment",
"value" : "right"
}
]
Any thoughts?
Using the Gson library, you can map the json in a Java object.
So, you have to create a class like this:
public class MyObject{
private String formatType;
private String value;
//Constuctors, Getter and Setter...
//.....
//.....
}
After, using the method fromJson you can create an array of MyObject.
Gson gson = new Gson();
MyObject[] array = gson.fromJson(new FileReader("file.json"), MyObject[].class);
You can also use json4s library as shown next:
import org.json4s._
import org.json4s.jackson.JsonMethods._
val json = """{
"data":[
{
"formatType": "text",
"value": "bgyufcie huis huids hufhsduhfsl hd"
},
{
"formatType": "text size",
"value": 12
},
{
"formatType": "text alignment",
"value" : "right"
}
]
}"""
val parsed = parse(json)
val value = (parsed \ "data" \\ classOf[JObject]).filter(m => m("formatType") == "text alignment")(0)("value")
// value: Any = right
The filter (parsed \ "data" \\ classOf[JObject]) extracts all the items into a List of Map i.e:
List(
Map(formatType -> text, value -> bgyufcie huis huids hufhsduhfsl hd),
Map(formatType -> text size, value -> 12), Map(formatType -> text alignment, value -> right)
).
From those we apply the filter filter(m => m("formatType") == "text alignment") to retrieve the record that we really need.
Use Dijon FTW!
Here is a test that demonstrates how easily the "right" value can be found in samples like yours:
import com.github.pathikrit.dijon._
val json = parse(
"""{
|"data":[
| {
| "formatType": "text",
| "value": "bgyufcie huis huids hufhsduhfsl hd"
| },
| {
| "formatType": "text size",
| "value": 12
| },
| {
| "formatType": "text alignment",
| "value" : "right"
| }
|]
|}""".stripMargin)
assert(json.data.toSeq.collect {
case obj if obj.formatType == "text alignment" => obj.value
}.head == "right")
I would use the Jackson library, it is very helpful for parsing JSON. You can read the JSON using an ObjectMapper.
Here is a full tutorial to get you started: https://www.mkyong.com/java/jackson-how-to-parse-json/
create a multiline JSON string, then parse that string directly into a Scala object, use the net.liftweb package to solve this.
import net.liftweb.json._
object SarahEmailPluginConfigTest {
implicit val formats = DefaultFormats
case class Mailserver(url: String, username: String, password: String)
val json = parse(
"""
{
"url": "imap.yahoo.com",
"username": "myusername",
"password": "mypassword"
}
"""
)
def main(args: Array[String]) {
val m = json.extract[Mailserver]
println(m.url)
println(m.username)
println(m.password)
}
}
https://alvinalexander.com/scala/simple-scala-lift-json-example-lift-framework
Reference link

Transforming JSON with Circe

Suppose I have some JSON data like this:
{
"data": {
"title": "example input",
"someBoolean": false,
"innerData": {
"innerString": "input inner string",
"innerBoolean": true,
"innerCollection": [1,2,3,4,5]
},
"collection": [6,7,8,9,0]
}
}
And I want to flatten it a bit and transform or remove some fields, to get the following result:
{
"data": {
"ttl": "example input",
"bool": false,
"collection": [6,7,8,9,0],
"innerCollection": [1,2,3,4,5]
}
}
How can I do this with Circe?
(Note that I'm asking this as a FAQ since similar questions often come up in the Circe Gitter channel. This specific example is from a question asked there yesterday.)
I've sometimes said that Circe is primarily a library for encoding and decoding JSON, not for transforming JSON values, and in general I'd recommend mapping to Scala types and then defining relationships between those (as Andriy Plokhotnyuk suggests here), but for many cases writing transformations with cursors works just fine, and in my view this kind of thing is one of them.
Here's how I'd implement this transformation:
import io.circe.{DecodingFailure, Json, JsonObject}
import io.circe.syntax._
def transform(in: Json): Either[DecodingFailure, Json] = {
val someBoolean = in.hcursor.downField("data").downField("someBoolean")
val innerData = someBoolean.delete.downField("innerData")
for {
boolean <- someBoolean.as[Json]
collection <- innerData.get[Json]("innerCollection")
obj <- innerData.delete.up.as[JsonObject]
} yield Json.fromJsonObject(
obj.add("boolean", boolean).add("collection", collection)
)
}
And then:
val Right(json) = io.circe.jawn.parse(
"""{
"data": {
"title": "example input",
"someBoolean": false,
"innerData": {
"innerString": "input inner string",
"innerBoolean": true,
"innerCollection": [1,2,3]
},
"collection": [6,7,8]
}
}"""
)
And:
scala> transform(json)
res1: Either[io.circe.DecodingFailure,io.circe.Json] =
Right({
"data" : {
"title" : "example input",
"collection" : [
6,
7,
8
]
},
"boolean" : false,
"collection" : [
1,
2,
3
]
})
If you look at it the right way, our transform method kind of resembles a decoder, and we can actually write it as one (although I'd definitely recommend not making it implicit):
import io.circe.{Decoder, Json, JsonObject}
import io.circe.syntax._
val transformData: Decoder[Json] = { c =>
val someBoolean = c.downField("data").downField("someBoolean")
val innerData = someBoolean.delete.downField("innerData")
(
innerData.delete.up.as[JsonObject],
someBoolean.as[Json],
innerData.get[Json]("innerCollection")
).mapN(_.add("boolean", _).add("collection", _)).map(Json.fromJsonObject)
}
This can be convenient in some situations where you want to perform the transformation as part of a pipeline that expects a decoder:
scala> io.circe.jawn.decode(myJsonString)(transformData)
res2: Either[io.circe.Error,io.circe.Json] =
Right({
"data" : {
"title" : "example input",
"collection" : [ ...
This is also potentially confusing, though, and I've thought about adding some kind of Transformation type to Circe that would encapsulate transformations like this without questionably repurposing the Decoder type class.
One nice thing about both the transform method and this decoder is that if the input data doesn't have the expected shape, the resulting error will include a history that points to the problem.

Scala play JSON, lookup and match defined field holding null value

I have the following Json block that I have returned as a JsObject
{
"first_block": [
{
"name": "demo",
"description": "first demo description"
}
],
"second_block": [
{
"name": "second_demo",
"description": "second demo description",
"nested_second": [
{
"name": "bob",
"value": null
},
{
"name": "john",
"value": null
}
]
}
]
}
From this, I want to return a list of all the possible values I could have in the second block, nested array for name and value. so with the example above
List([bob,null],[john,null]) or something along those lines.
The issue I am having is with the value section understanding null values. I've tried to match against it and return a string "null" but I can't get it to match on Null values.
What would be the best way for me to return back the name and values in the nested_second array.
I've tried using case classes and readAsNullable with no luck, and my latest attempt has gone along these lines:
val secondBlock = (jsObj \ "second_block").as[List[JsValue]]
secondBlock.foreach(nested_block => {
val nestedBlock = (nested_block \ "nested_second").as[List[JsValue]]
nestedBlock.foreach(value => {
val name = (value \ "name").as[String] //always a string
var convertedValue = ""
val replacement_value = value \ "value"
replacement_value match {
case JsDefined(null) => convertedValue = "null"
case _ => convertedValue = replacement_value.as[String]
}
println(name)
println(convertedValue)
})
}
)
It seems convertedValue returns as 'JsDefined(null)' regardless and I'm sure the way I'm doing it is horrifically bad.
Replace JsDefined(null) with JsDefined(JsNull).
You probably got confused, because println(JsDefined(JsNull)) prints as JsDefined(null). But that is not, how null value of a JSON field is represented. null is represented as case object JsNull. This is just a good API design, where possible cases are represented with a hierarchy of classes:
With play-json I use always case-classes!
I simplified your problem to the essence:
import play.api.libs.json._
val jsonStr = """[
{
"name": "bob",
"value": null
},
{
"name": "john",
"value": "aValue"
},
{
"name": "john",
"value": null
}
]"""
Define a case class
case class Element(name: String, value: Option[String])
Add a formatter in the companion object:
object Element {
implicit val jsonFormat: Format[Element] = Json.format[Element]
}
An use validate:
Json.parse(jsonStr).validate[Seq[Element]] match {
case JsSuccess(elems, _) => println(elems)
case other => println(s"Handle exception $other")
}
This returns: List(Element(bob,None), Element(john,Some(aValue)), Element(john,None))
Now you can do whatever you want with the values.