I am working with the Scala Circe library. It seems very useful and I want to get better at using it.
One example I have is the following.
Consider the following code:
import io.circe.syntax._
import io.circe.{Json, Encoder}
import io.circe.generic.auto._
sealed trait JsonComponent
case class DataSequences(values: Map[String, List[Int]]) extends JsonComponent
case class DataField(data: List[DataSequences], `type`: String) extends JsonComponent
object Example extends App {
implicit val encodeDataSequences: Encoder[DataSequences] = new Encoder[DataSequences] {
final def apply(sequence: DataSequences): Json = sequence.values.asJson
}
val x = new DataSequences(Map("x" -> List(1, 2, 3), "y" -> List(1, 2, 4)))
val l = List(DataField(List(x, x), "abc"), DataField(List(x, x), "cde")).asJson
println(l)
}
This gives the following output:
[
{
"data" : [
{
"x" : [
1,
2,
3
],
"y" : [
1,
2,
4
]
},
{
"x" : [
1,
2,
3
],
"y" : [
1,
2,
4
]
}
],
"type" : "abc"
}
]
However, if I comment out the encodeDataSequences encoder definition, I get the following instead:
[
{
"data" : [
{
"x" : [
1,
2,
3
],
"y" : [
1,
2,
4
]
},
{
"x" : [
1,
2,
3
],
"y" : [
1,
2,
4
]
}
],
"type" : "abc"
}
]
So now this "values" appears. I do not want the "values" field to show up. I am not sure how the implicit is shaping the Json under the hood, and if someone could highlight what's going on that would be appreciated.
In addition, as a gernal thing, am I writing idiomatic Circe code using that implicit encoder, and if so is there a better way to do what I want?
Related
Suppose I have a JSON object like this:
{
"foo": true,
"bar": {
"baz": 1,
"qux": {
"msg": "hello world",
"wow": [null]
}
}
}
And I want to flatten it recursively to a single layer, with the keys merged with an underscore:
{
"foo": true,
"bar_baz": 1,
"baz_qux_msg": "hello world",
"baz_qux_wow": [null]
}
How can I do this with Circe?
(Note: this is another FAQ from the Circe Gitter channel.)
You can do this without too much pain in Circe with a recursive method:
import io.circe.Json
def flatten(combineKeys: (String, String) => String)(value: Json): Json = {
def flattenToFields(value: Json): Option[Iterable[(String, Json)]] =
value.asObject.map(
_.toIterable.flatMap {
case (k, v) => flattenToFields(v) match {
case None => List(k -> v)
case Some(fields) => fields.map {
case (innerK, innerV) => combineKeys(k, innerK) -> innerV
}
}
}
)
flattenToFields(value).fold(value)(Json.fromFields)
}
Here our internal flattenToFields method takes each JSON value and either returns None if it's a non-JSON object value, as a signal that that field doesn't need flattening, or a Some containing a sequence of flattened fields in the case of a JSON object.
If we have a JSON value like this:
val Right(doc) = io.circe.jawn.parse("""{
"foo": true,
"bar": {
"baz": 1,
"qux": {
"msg": "hello world",
"wow": [null]
}
}
}""")
We can verify that flatten does what we want like this:
scala> flatten(_ + "_" + _)(doc)
res1: io.circe.Json =
{
"foo" : true,
"bar_baz" : 1,
"bar_qux_msg" : "hello world",
"bar_qux_wow" : [
null
]
}
Note that flattenToFields is not tail recursive, and will overflow the stack for deeply-nested JSON objects, but probably not until you're several thousand levels deep, so it's unlikely to be an issue in practice. You could make it tail recursive without too much trouble, but at the expense of additional overhead for the common cases where you only have a few layers of nesting.
I propose a variation of the solution by Travis Brown. The variation concerns objects in the JSON lists, i.e. how to handle
{
"foo": true,
"bar": {
"baz": 1,
"qux": {
"msg": "hello world",
"wow": [{"x": 1, "y": 2}, {"x": 3, "y": 4}]
}
}
}
One possible solution for recursively handling objects in lists is the following implementation, where the position in the list is taken as an additional key part
def flattenDeep(combineKeys: (String, String) => String)(value: Json): Json = {
def flattenToFields(value: Json): Option[Iterable[(String, Json)]] = {
value.fold(
jsonNull = None,
jsonNumber = _ => None,
jsonString = _ => None,
jsonBoolean = _ => None,
jsonObject = { obj =>
val fields = obj.toIterable.flatMap {
case (field, json) =>
flattenToFields(json).fold(Iterable(field -> json)) {
_.map {
case (innerField, innerJson) =>
combineKeys(field, innerField) -> innerJson
}
}
}
Some(fields)
},
jsonArray = { array =>
val fields = array.zipWithIndex.flatMap {
case (json, index) =>
flattenToFields(json).fold(Iterable(index.toString -> json)) {
_.map {
case (innerField, innerJson) =>
combineKeys(index.toString, innerField) -> innerJson
}
}
}
Some(fields)
}
)
}
flattenToFields(value).fold(value)(Json.fromFields)
}
With this implementation the above example is flattened to:
{
"foo" : true,
"bar_baz" : 1,
"bar_qux_msg" : "hello world",
"bar_qux_wow_0_x" : 1,
"bar_qux_wow_0_y" : 2,
"bar_qux_wow_1_x" : 3,
"bar_qux_wow_1_y" : 4
}
For even deeper nested structures one still gets a flat representation, e.g.
{
"foo": true,
"bar": {
"baz": 1,
"qux": {
"msg": "hello world",
"wow": [
{
"x": 1,
"y": 2
},
{
"x": 3,
"y": 4
}
],
"deeper": [
{
"alpha": {
"h": 12,
"m": 1
},
"beta": [ "a", "b", "c" ]
},
{
"alpha": {
"h": 21,
"m": 0
},
"beta": [ "z" ]
}
]
}
}
}
will be flattened into
{
"foo" : true,
"bar_baz" : 1,
"bar_qux_msg" : "hello world",
"bar_qux_wow_0_x" : 1,
"bar_qux_wow_0_y" : 2,
"bar_qux_wow_1_x" : 3,
"bar_qux_wow_1_y" : 4,
"bar_qux_deeper_0_alpha_h" : 12,
"bar_qux_deeper_0_alpha_m" : 1,
"bar_qux_deeper_0_beta_0" : "a",
"bar_qux_deeper_0_beta_1" : "b",
"bar_qux_deeper_0_beta_2" : "c",
"bar_qux_deeper_1_alpha_h" : 21,
"bar_qux_deeper_1_alpha_m" : 0,
"bar_qux_deeper_1_beta_0" : "z"
}
How can I read a "complex" json using Klaxon?
I'm trying to use klaxon's stream api as documentation say .
I'm using the beginObject method. If I use a json as given in the example everything is fine
val objectString = """{
"name" : "Joe",
"age" : 23,
"flag" : true,
"array" : [1, 3],
"obj1" : { "a" : 1, "b" : 2 }
}"""
But if I try to parse a json with a nested object as the following example I get the following error: "Expected a name but got LEFT_BRACE"
val objectString = """{
"name" : "Joe",
"age" : 23,
"flag" : true,
"array" : [1, 3],
"obj1" : {
"hello": { "a" : 1, "b" : 2 }
}
}"""
I don't see any reported issues in the github repo, so I wonder if there is a way to get this working.
cheers
Ok so I checked out the source, and it seems that nextObject() assumes that you are dealing with a simple key value object, where values are not objects.
Anyway, there is another way to parse a JSON of the format you specified, like below:
val objectString = """{
"name" : "Joe",
"age" : 23,
"flag" : true,
"array" : [1, 2, 3],
"obj1" : { "hello": {"a" : 1, "b" : 2 } }
}"""
class ABObj(val a: Int, val b: Int)
class HelloObj(val hello: ABObj)
class Obj(val name: String, val age: Int, val flag: Boolean, val array: List<Any>, val obj1: HelloObj)
val r = Klaxon().parse<Obj>(objectString)
// r is your parsed object
print(r!!.name) // prints Joe
print(r!!.array) // prints [1, 2, 3]
The classes I created are like below:
ABObj represents this JSON:
{"a": 1, "b": 2}
HelloObj represents this JSON:
{"hello": {"a": 1, "b": 2}}
And finally, Obj refers to the top level object.
Hope this helps.
With the Streaming API, you are in charge of parsing objects and arrays yourself, so when you encounter the name "obj1", you need to start a new beginObject() { ... } and parse that object there.
The documentation that you linked does exactly that:
while (reader.hasNext()) {
val readName = reader.nextName()
when (readName) {
"name" -> name = reader.nextString()
"age" -> age = reader.nextInt()
"flag" -> flag = reader.nextBoolean()
"array" -> array = reader.nextArray()
"obj1" -> obj1 = reader.nextObject()
I have some json that includes some fields that are being flattened into a bson-ish format as in {"foo.bar" : "bash"}. I'd like to transform this to the following representation {"foo" : { "bar" : "bash"}} and wondering where in circe I'd do such an operation. Complicating the problem is that there could be multiple such fields that need to be properly merged, e.g. {"foo.bar" : "a", "foo.bash" : "b", "foo.baz" : "c"} -> {"foo" : { "bar" : "a", "bash" : "b", "baz" : "c"}}.
Here's a quick implementation:
import io.circe.Json
val Dotted = "([^\\.]*)\\.(.*)".r
def expandDotted(j: Json): Json = j.arrayOrObject(
j,
js => Json.fromValues(js.map(expandDotted)),
_.toList.map {
case (Dotted(k, rest), v) => Json.obj(k -> expandDotted(Json.obj(rest -> v)))
case (k, v) => Json.obj(k -> expandDotted(v))
}.reduceOption(_.deepMerge(_)).getOrElse(Json.obj())
)
I haven't really used or tested it in detail, but it seems to work:
scala> import io.circe.literal._
import io.circe.literal._
scala> val j1 = json"""{"foo.bar" : "a", "foo.bash" : "b", "foo.baz" : "c"}"""
j1: io.circe.Json =
{
"foo.bar" : "a",
"foo.bash" : "b",
"foo.baz" : "c"
}
scala> expandDotted(j1)
res1: io.circe.Json =
{
"foo" : {
"baz" : "c",
"bash" : "b",
"bar" : "a"
}
}
And with deeper nesting:
scala> expandDotted(json"""{ "x.y.z": true, "a.b": { "c": 1 } }""")
res2: io.circe.Json =
{
"a" : {
"b" : {
"c" : 1
}
},
"x" : {
"y" : {
"z" : true
}
}
}
And just to confirm that it doesn't mess with undotted keys:
scala> expandDotted(json"""{ "a.b": true, "x": 1 }""").noSpaces
res3: String = {"x":1,"a":{"b":true}}
Note that in the case of "collisions" (paths that lead to both JSON objects and non-object JSON values, or to multiple non-object values), the behavior is that of Json#deepMerge:
scala> expandDotted(json"""{ "a.b": true, "a": 1 }""").noSpaces
res4: String = {"a":1}
scala> expandDotted(json"""{ "a": 1, "a.b": true }""").noSpaces
res5: String = {"a":{"b":true}}
…which is probably what you'd want, but you could also have it fail in these cases, or not expand the colliding path, or do pretty much any other thing you can think of.
I have an RDD of type RDD[(String, List[String])].
Example:
(FRUIT, List(Apple,Banana,Mango))
(VEGETABLE, List(Potato,Tomato))
I want to convert the above output to json object like below.
{
"categories": [
{
"name": "FRUIT",
"nodes": [
{
"name": "Apple",
"isInTopList": false
},
{
"name": "Banana",
"isInTopList": false
},
{
"name": "Mango",
"isInTopList": false
}
]
},
{
"name": "VEGETABLE",
"nodes": [
{
"name": "POTATO",
"isInTopList": false
},
{
"name": "TOMATO",
"isInTopList": false
},
]
}
]
}
Please suggest the best possible way to do it.
NOTE: "isInTopList": false is always constant and has to be there with every item in the jsonobject.
First I used the following code to reproduce the scenario that you mentioned:
val sampleArray = Array(
("FRUIT", List("Apple", "Banana", "Mango")),
("VEGETABLE", List("Potato", "Tomato")))
val sampleRdd = sc.parallelize(sampleArray)
sampleRdd.foreach(println) // Printing the result
Now, I am using json4s Scala library to convert this RDD into the JSON structure that you requested:
import org.json4s.native.JsonMethods._
import org.json4s.JsonDSL.WithDouble._
val json = "categories" -> sampleRdd.collect().toList.map{
case (name, nodes) =>
("name", name) ~
("nodes", nodes.map{
name => ("name", name)
})
}
println(compact(render(json))) // Printing the rendered JSON
The result is:
{"categories":[{"name":"FRUIT","nodes":[{"name":"Apple"},{"name":"Banana"},{"name":"Mango"}]},{"name":"VEGETABLE","nodes":[{"name":"Potato"},{"name":"Tomato"}]}]}
Since you want a single JSON for you entire RDD, I would start by doing Rdd.collect. Be careful that your set fits in memory, as this will move the data back to the driver.
To get the json, just use a library to traverse your objects. I like Json4s due to its simple internal structure and practical, clean operators. Here is a sample from their website that shows how to traverse nested structures (in particular, lists):
object JsonExample extends App {
import org.json4s._
import org.json4s.JsonDSL._
import org.json4s.jackson.JsonMethods._
case class Winner(id: Long, numbers: List[Int])
case class Lotto(id: Long, winningNumbers: List[Int], winners: List[Winner], drawDate: Option[java.util.Date])
val winners = List(Winner(23, List(2, 45, 34, 23, 3, 5)), Winner(54, List(52, 3, 12, 11, 18, 22)))
val lotto = Lotto(5, List(2, 45, 34, 23, 7, 5, 3), winners, None)
val json =
("lotto" ->
("lotto-id" -> lotto.id) ~
("winning-numbers" -> lotto.winningNumbers) ~
("draw-date" -> lotto.drawDate.map(_.toString)) ~
("winners" ->
lotto.winners.map { w =>
(("winner-id" -> w.id) ~
("numbers" -> w.numbers))}))
println(compact(render(json)))
}
I've a List of a type which contains three fields (services, total, improved). When I convert it to JSON using Json.toJson(myList), I get the JSON in the following format:
[
{
"services":"S4",
"total":1,
"improved":1
},
{
"services":"S1",
"total":2,
"improved":1
},
{
"services":"S2",
"total":3,
"improved":2
}
]
Using JSON library in Play 2.x in Scala, how can I convert myList in the following JSON format?
[
{
"key" : "total",
"values" :[
{
"services" : "s1",
"value" : 2
},
"services" : "s2",
"value" : 3
{
"services" : "s4",
"value" : 1
}
]
},
{
"key" : "improved",
"values" :[
{
"services" : "s1",
"value" : 1
},
"services" : "s2",
"value" : 2
{
"services" : "s4",
"value" : 1
}
]
}
]
Thanks in advance.
Since you are already dealing with OO, I think you could wrap your List into a more specific object and convert it to JSON :
case class Foo(services: String, total: Int, improved: Int)
case class B(key: String, value: Int)
case class A(key: String, values: Seq[B] = Seq())
val myOriginalList = Seq(Foo("S4", 1, 1), Foo("S1", 2, 1), Foo("S2", 3, 2))
val transformedList = myOriginalList.foldLeft((A("total"), A("improved")))({ (res, x) =>
(res._1.copy(values = B(x.services, x.total) +: res._1.values),
res._2.copy(values = B(x.services, x.improved) +: res._2.values))
}).map({x => List(x._1, x._2)})
Json.toJson(transformedList)
One of the problem (or maybe not) using this solution, is that you don't resolve the Foo attributes dynamically.
You could also try yourself around Json Transformers : http://www.playframework.com/documentation/2.2.x/ScalaJsonTransformers
Here's a Scala newbie's solution (I'm sure it's not elegant):
I'm using a solution provided here: http://www.playframework.com/documentation/2.1.1/ScalaJson
//some data
case class PatientAggregateValues(total: Int, improved: Int)
val scoreMap = Map.empty[String, PatientAggregateValues]
scoreMap += ("S1" -> PatientAggregateValues(2, 1))
scoreMap += ("S2" -> PatientAggregateValues(3, 2))
scoreMap += ("S4" -> PatientAggregateValues(1, 1))
//main logic
val totalMap = scoreMap map { case (k,v) => Json.toJson(scala.collection.immutable.Map("service" -> Json.toJson(k), "value" -> Json.toJson(v.total))) } toSeq
val improvedMap = scoreMap map { case (k,v) => Json.toJson(scala.collection.immutable.Map("service" -> Json.toJson(k), "value" -> Json.toJson(v.improved))) } toSeq
Json.toJson(scala.collection.immutable.Map("total" -> totalMap, "improved" -> improvedMap))
Thanks.