I am using play.api.libs.json in Scala (2.12.8) to process some json objects. I have for example a JSON string that looks like:
{
"field1": null,
"field2": 23,
"field3": {
"subfield1": "a",
"subfield2": null
},
"field4": {
"subfield1": true,
"subfield2": {
"subsubfield1": null,
"subsubfield2": "45"
},
"field5": 3
}
}
And I want to filter out every null fields or subfields.
As explained here: Play: How to remove the fields without value from JSON and create a new JSON with them
Doing:
import play.api.libs.json.{ JsNull, JsObject, JsValue, Json }
val j = Json.parse(myJsonString).as[JsObject]
JsObject(j.fields.filterNot(t => withoutValue(t._2)))
def withoutValue(v: JsValue) = v match {
case JsNull => true
case _ => false
}
helps me remove the upper level fields: in my case, field1
But field3.subfield2 and field4.subfield2.subsubfield1 are still present. I want to remove them. Also I should mention that not every subfields can be null at once. Should this happen, I think we could just remove the upper level field. If field3.subfield1and field3.subfield2 are null, we can remove field3.
Any idea on how to do this neatly in Scala?
PS: the desired output is:
{
"field2": 23,
"field3": {
"subfield1": "a"
},
"field4": {
"subfield1": true,
"subfield2": {
"subsubfield2": "45"
},
"field5": 3
}
}
You need to do a recursive solution. For example:
def removeNulls(jsObject: JsObject): JsValue = {
JsObject(jsObject.fields.collect {
case (s, j: JsObject) =>
(s, removeNulls(j))
case other if (other._2 != JsNull) =>
other
})
}
Code run at Scastie. Output is as expected.
Related
How do I create a Json (Circe) looking like this:
{
"items": [{
"field1": "somevalue",
"field2": "somevalue2"
},
{
"field1": "abc",
"field2": "123abc"
}]
}
val result = Json.fromFields(List("items" -> ???))
You can do so using Circe's built in list typeclasses for encoding JSON. This code will work:
import io.circe.{Encoder, Json}
import io.circe.syntax._
case class Field(field1: String, field2: String)
object Field {
implicit val encodeFoo: Encoder[Field] = new Encoder[Field] {
final def apply(a: Field): Json = Json.obj(
("field1", Json.fromString(a.field1)),
("field2", Json.fromString(a.field2))
)
}
}
class Encoding(items: List[Field]) {
def getJson: Json = {
Json.obj(
(
"items",
items.asJson
)
)
}
}
If we instantiate an "Encoding" class and call getJson it will give you back the desired JSON. It works because with circe all you need to do to encode a list is provide an encoder for whatever is inside the list. Thus, if we provide an encoder for Field it will encode it inside a list when we call asJson on it.
If we run this:
val items = new Encoding(List(Field("jf", "fj"), Field("jfl", "fjl")))
println(items.getJson)
we get:
{
"items" : [
{
"field1" : "jf",
"field2" : "fj"
},
{
"field1" : "jfl",
"field2" : "fjl"
}
]
}
Suppose I have a JSON object like this:
{
"foo": true,
"bar": {
"baz": 1,
"qux": {
"msg": "hello world",
"wow": [null]
}
}
}
And I want to flatten it recursively to a single layer, with the keys merged with an underscore:
{
"foo": true,
"bar_baz": 1,
"baz_qux_msg": "hello world",
"baz_qux_wow": [null]
}
How can I do this with Circe?
(Note: this is another FAQ from the Circe Gitter channel.)
You can do this without too much pain in Circe with a recursive method:
import io.circe.Json
def flatten(combineKeys: (String, String) => String)(value: Json): Json = {
def flattenToFields(value: Json): Option[Iterable[(String, Json)]] =
value.asObject.map(
_.toIterable.flatMap {
case (k, v) => flattenToFields(v) match {
case None => List(k -> v)
case Some(fields) => fields.map {
case (innerK, innerV) => combineKeys(k, innerK) -> innerV
}
}
}
)
flattenToFields(value).fold(value)(Json.fromFields)
}
Here our internal flattenToFields method takes each JSON value and either returns None if it's a non-JSON object value, as a signal that that field doesn't need flattening, or a Some containing a sequence of flattened fields in the case of a JSON object.
If we have a JSON value like this:
val Right(doc) = io.circe.jawn.parse("""{
"foo": true,
"bar": {
"baz": 1,
"qux": {
"msg": "hello world",
"wow": [null]
}
}
}""")
We can verify that flatten does what we want like this:
scala> flatten(_ + "_" + _)(doc)
res1: io.circe.Json =
{
"foo" : true,
"bar_baz" : 1,
"bar_qux_msg" : "hello world",
"bar_qux_wow" : [
null
]
}
Note that flattenToFields is not tail recursive, and will overflow the stack for deeply-nested JSON objects, but probably not until you're several thousand levels deep, so it's unlikely to be an issue in practice. You could make it tail recursive without too much trouble, but at the expense of additional overhead for the common cases where you only have a few layers of nesting.
I propose a variation of the solution by Travis Brown. The variation concerns objects in the JSON lists, i.e. how to handle
{
"foo": true,
"bar": {
"baz": 1,
"qux": {
"msg": "hello world",
"wow": [{"x": 1, "y": 2}, {"x": 3, "y": 4}]
}
}
}
One possible solution for recursively handling objects in lists is the following implementation, where the position in the list is taken as an additional key part
def flattenDeep(combineKeys: (String, String) => String)(value: Json): Json = {
def flattenToFields(value: Json): Option[Iterable[(String, Json)]] = {
value.fold(
jsonNull = None,
jsonNumber = _ => None,
jsonString = _ => None,
jsonBoolean = _ => None,
jsonObject = { obj =>
val fields = obj.toIterable.flatMap {
case (field, json) =>
flattenToFields(json).fold(Iterable(field -> json)) {
_.map {
case (innerField, innerJson) =>
combineKeys(field, innerField) -> innerJson
}
}
}
Some(fields)
},
jsonArray = { array =>
val fields = array.zipWithIndex.flatMap {
case (json, index) =>
flattenToFields(json).fold(Iterable(index.toString -> json)) {
_.map {
case (innerField, innerJson) =>
combineKeys(index.toString, innerField) -> innerJson
}
}
}
Some(fields)
}
)
}
flattenToFields(value).fold(value)(Json.fromFields)
}
With this implementation the above example is flattened to:
{
"foo" : true,
"bar_baz" : 1,
"bar_qux_msg" : "hello world",
"bar_qux_wow_0_x" : 1,
"bar_qux_wow_0_y" : 2,
"bar_qux_wow_1_x" : 3,
"bar_qux_wow_1_y" : 4
}
For even deeper nested structures one still gets a flat representation, e.g.
{
"foo": true,
"bar": {
"baz": 1,
"qux": {
"msg": "hello world",
"wow": [
{
"x": 1,
"y": 2
},
{
"x": 3,
"y": 4
}
],
"deeper": [
{
"alpha": {
"h": 12,
"m": 1
},
"beta": [ "a", "b", "c" ]
},
{
"alpha": {
"h": 21,
"m": 0
},
"beta": [ "z" ]
}
]
}
}
}
will be flattened into
{
"foo" : true,
"bar_baz" : 1,
"bar_qux_msg" : "hello world",
"bar_qux_wow_0_x" : 1,
"bar_qux_wow_0_y" : 2,
"bar_qux_wow_1_x" : 3,
"bar_qux_wow_1_y" : 4,
"bar_qux_deeper_0_alpha_h" : 12,
"bar_qux_deeper_0_alpha_m" : 1,
"bar_qux_deeper_0_beta_0" : "a",
"bar_qux_deeper_0_beta_1" : "b",
"bar_qux_deeper_0_beta_2" : "c",
"bar_qux_deeper_1_alpha_h" : 21,
"bar_qux_deeper_1_alpha_m" : 0,
"bar_qux_deeper_1_beta_0" : "z"
}
I have a json file , which has some keys and values. I need to parse the Json and print the keys and their values. For example, the json file is like below. I want to print this as Keys and values
{
"Parcer":[
{
"key":"0203",
"value":{
"Encryption":
{
"enabled":"yes",
"encryption_type":"base64",
"key":"334848484",
"return":"name"
}
}
},
{
"key":"0405",
"value":{
"Encryption":
{
"enabled":"yes",
"encryption_type":"base64",
"key":"334848484",
"return":"none"
},
"Parcer":[
{
"key":"0102",
"value":"humidity"
},
{
"key":"0304",
"value":{
"Encryption":{
"enabled":"yes",
"encryption_type":"SHA1",
"key":"1211212",
"return":"none"
}
}
}
]
}
}],
}```
The easiest way is to create a case class, like:
case class MyObj(header:String, value: Seq[Map[String, String]])
Then you just need to add one line for marshalling, like:
import play.api.libs.json._
object MyObj {
implicit val jsonFormat: OFormat[MyObj] = Json.format[MyObj]
}
Now you get a nice case class that you can work with:
val json =
Json.parse(
"""{
"header" : "header value",
"value" : [
{
"a" : "a_val",
"b" : "b_val",
"c" : "c_val"
},
{
"a" : "a_val",
"b" : "b_val",
"c" : "c_val"
}
]
}""")
Here an example how to retrieve all "a".
json.validate[MyObj] match {
case JsSuccess(myObj, _) =>
val allAs =myObj.value.flatMap(m => m.get("a").toSeq)
println(allAs) // >> Vector(a_val, a_val)
case e:JsError => // handle error
}
This gives you:
json.validate[MyObj] returns JsSuccess(MyObj(header value,Vector(Map(a -> a_val, b -> b_val, c -> c_val), Map(a -> a_val, b -> b_val, c -> c_val))),)
The println returns: Vector(a_val, a_val)
This is described here in the Documentation: JSON automated mapping
I'm getting a JSON object over the network, as a String. I'm then using Circe to parse it. I want to add a handful of fields to it, and then pass it on downstream.
Almost all of that works.
The problem is that my "adding" is really "overwriting". That's actually ok, as long as I add an empty object first. How can I add such an empty object?
So looking at the code below, I am overwriting "sometimes_empty:{}" and it works. But because sometimes_empty is not always empty, it results in some data loss. I'd like to add a field like: "custom:{}" and then ovewrite the value of custom with my existing code.
Two StackOverflow posts were helpful. One worked, but wasn't quite what I was looking for. The other I couldn't get to work.
1: Modifying a JSON array in Scala with circe
2: Adding field to a json using Circe
val js: String = """
{
"id": "19",
"type": "Party",
"field": {
"id": 1482,
"name": "Anne Party",
"url": "https"
},
"sometimes_empty": {
},
"bool": true,
"timestamp": "2018-12-18T11:39:18Z"
}
"""
val newJson = parse(js).toOption
.flatMap { doc =>
doc.hcursor
.downField("sometimes_empty")
.withFocus(_ =>
Json.fromFields(
Seq(
("myUrl", Json.fromString(myUrl)),
("valueZ", Json.fromString(valueZ)),
("valueQ", Json.fromString(valueQ)),
("balloons", Json.fromString(balloons))
)
)
)
.top
}
newJson match {
case Some(v) => return v.toString
case None => println("Failure!")
}
We need to do a couple of things. First, we need to zoom in on the specific property we want to update, if it doesn't exist, we'll create a new empty one. Then, we turn the zoomed in property in the form of a Json into JsonObject in order to be able to modify it using the +: method. Once we've done that, we need to take the updated property and re-introduce it in the original parsed JSON to get the complete result:
import io.circe.{Json, JsonObject, parser}
import io.circe.syntax._
object JsonTest {
def main(args: Array[String]): Unit = {
val js: String =
"""
|{
| "id": "19",
| "type": "Party",
| "field": {
| "id": 1482,
| "name": "Anne Party",
| "url": "https"
| },
| "bool": true,
| "timestamp": "2018-12-18T11:39:18Z"
|}
""".stripMargin
val maybeAppendedJson =
for {
json <- parser.parse(js).toOption
sometimesEmpty <- json.hcursor
.downField("sometimes_empty")
.focus
.orElse(Option(Json.fromJsonObject(JsonObject.empty)))
jsonObject <- json.asObject
emptyFieldJson <- sometimesEmpty.asObject
appendedField = emptyFieldJson.+:("added", Json.fromBoolean(true))
res = jsonObject.+:("sometimes_empty", appendedField.asJson)
} yield res
maybeAppendedJson.foreach(obj => println(obj.asJson.spaces2))
}
}
Yields:
{
"id" : "19",
"type" : "Party",
"field" : {
"id" : 1482,
"name" : "Anne Party",
"url" : "https"
},
"sometimes_empty" : {
"added" : true,
"someProperty" : true
},
"bool" : true,
"timestamp" : "2018-12-18T11:39:18Z"
}
I have the following Json block that I have returned as a JsObject
{
"first_block": [
{
"name": "demo",
"description": "first demo description"
}
],
"second_block": [
{
"name": "second_demo",
"description": "second demo description",
"nested_second": [
{
"name": "bob",
"value": null
},
{
"name": "john",
"value": null
}
]
}
]
}
From this, I want to return a list of all the possible values I could have in the second block, nested array for name and value. so with the example above
List([bob,null],[john,null]) or something along those lines.
The issue I am having is with the value section understanding null values. I've tried to match against it and return a string "null" but I can't get it to match on Null values.
What would be the best way for me to return back the name and values in the nested_second array.
I've tried using case classes and readAsNullable with no luck, and my latest attempt has gone along these lines:
val secondBlock = (jsObj \ "second_block").as[List[JsValue]]
secondBlock.foreach(nested_block => {
val nestedBlock = (nested_block \ "nested_second").as[List[JsValue]]
nestedBlock.foreach(value => {
val name = (value \ "name").as[String] //always a string
var convertedValue = ""
val replacement_value = value \ "value"
replacement_value match {
case JsDefined(null) => convertedValue = "null"
case _ => convertedValue = replacement_value.as[String]
}
println(name)
println(convertedValue)
})
}
)
It seems convertedValue returns as 'JsDefined(null)' regardless and I'm sure the way I'm doing it is horrifically bad.
Replace JsDefined(null) with JsDefined(JsNull).
You probably got confused, because println(JsDefined(JsNull)) prints as JsDefined(null). But that is not, how null value of a JSON field is represented. null is represented as case object JsNull. This is just a good API design, where possible cases are represented with a hierarchy of classes:
With play-json I use always case-classes!
I simplified your problem to the essence:
import play.api.libs.json._
val jsonStr = """[
{
"name": "bob",
"value": null
},
{
"name": "john",
"value": "aValue"
},
{
"name": "john",
"value": null
}
]"""
Define a case class
case class Element(name: String, value: Option[String])
Add a formatter in the companion object:
object Element {
implicit val jsonFormat: Format[Element] = Json.format[Element]
}
An use validate:
Json.parse(jsonStr).validate[Seq[Element]] match {
case JsSuccess(elems, _) => println(elems)
case other => println(s"Handle exception $other")
}
This returns: List(Element(bob,None), Element(john,Some(aValue)), Element(john,None))
Now you can do whatever you want with the values.