How to transform a named JsArray into a JsObject with JsZipper - json

Let's assume either of the following two JSON snippets:
{ "include": ["field1", "field2", "fieldN"] }
{ "exclude": ["field1", "field2", "fieldN"] }
I need to transform the include array like this...
{ "field1": 1, "field2": 1, "fieldN": 1 }
... and the exclude array like this:
{ "field1": 0, "field2": 0, "fieldN": 0 }
[Just for your info: I need to transform input JSON into Mongo's projections.]
Here below is my current solution – I've implemented it as a JsValue extension:
object testTypeExtensions {
implicit class TestJsExtensions(val json: JsValue) extends AnyVal {
def toProjection: JsValue = {
if (json \\ "include" nonEmpty)
JsObject(for (field <- (json \ "include").as[List[JsString]])
yield (field.value, JsNumber(1)))
else if (json \\ "exclude" nonEmpty)
JsObject(for (field <- (json \ "exclude").as[List[JsString]])
yield (field.value, JsNumber(0)))
else Json.obj()
}
}
}
The code above works...
scala> val p = Json.obj("exclude" -> Json.arr("field1", "field2"))
p: play.api.libs.json.JsObject = {"exclude":["field1","field2"]}
scala> p.toProjection
res12: play.api.libs.json.JsObject = {"field1":0,"field2":0}
... but I'm sure it could be written much better with JsZipper.
Furthermore it is not very flexible since it only manages the include and exclude keys, whereas I want to also manage other similar cases like sorting objects:
{ "asc": ["field1", "field2"] }
{ "desc": ["field1", "field2"] }
... transformed into...
{ "field1": 1, "field2": 1 }
... and
{ "field1": -1, "field2": -1 }
That said, what I've in mind is a generic method that manages any kind of named JSON array like:
object testypeExtensions {
implicit class TempJsExtensions(val json: JsValue) extends AnyVal {
def namedArrayToObject(keys: String*): JsValue = {
// how to implement it, possibly with JsZipper
}
}
}
The namedArrayToObject method should search for the specified keys in the current JSON and generate an object for the first match like the ones I described at the beginning of this post, possibly with JsZipper. Here is a simulation of the expected results.
Search for exclude and include and return the first match as a JsObject:
scala> val p = Json.obj("exclude" -> Json.arr("field1", "field2"))
scala> p.namedArrayToObject("exclude", "include")
res12: play.api.libs.json.JsObject = {"field1":0,"field2":0}
Same as before... but now input JSON contains include instead of exclude:
scala> val p = Json.obj("include" -> Json.arr("field1", "field2"))
scala> p.namedArrayToObject("exclude", "include")
res12: play.api.libs.json.JsObject = {"field1":1,"field2":1}
Search for asc and desc and return the first match as a JsObject:
scala> val p = Json.obj("desc" -> Json.arr("field1", "field2"))
scala> p.namedArrayToObject("asc", "desc")
res12: play.api.libs.json.JsObject = {"field1":-1,"field2":-1}
... and so on.
If there is no match, namedArrayToObject should return an empty JsObject. Any suggestion on how to implement this in the right way would be very appreciated.

You can do this pretty straightforwardly with JSON transformations:
import play.api.libs.json._
def toObj(value: Int) = Reads.of[List[String]].map(
keys => Json.toJson(keys.map(_ -> value).toMap)
)
val transformation =
(__ \ 'include).json.update(toObj(1)) andThen
(__ \ 'exclude).json.update(toObj(0))
We can define an example object and apply our transformation:
val example = Json.parse("""{
"include": ["field1", "field2", "field3"],
"exclude": ["field4", "field5", "field6"]
}""")
val transformed = example.transform(transformation)
And then:
scala> transformed.foreach(Json.prettyPrint _ andThen println)
{
"include" : {
"field1" : 1,
"field2" : 1,
"field3" : 1
},
"exclude" : {
"field4" : 0,
"field5" : 0,
"field6" : 0
}
}
This doesn't exactly match your desired API, but it should be easily adaptable, and I'd suggest staying away from the implicit class business, anyway—it's much less composable and makes handling invalid input less elegant.

Travis helped me to find the way... and here below eventually is my solution that does exactly what I was looking for:
object testExtensions {
implicit class testJsExtensions(val json: JsValue) extends AnyVal {
def toParams(pairs: (String, Int)*): JsValue = {
for (pair <- pairs) { json.getOpt(__ \ pair._1).map { values =>
JsObject(for (field <- values.as[List[JsString]])
yield (field.value, JsNumber(pair._2)))
} match {
case Some(params) => return params
case _ =>
}}
Json.obj()
}
}
}
scala> example.toParams(("include", 1), ("exclude", 0))
res63: play.api.libs.json.JsValue = {"field1":1,"field2":1,"field3":1}
scala> example.toParams(("exclude", 0))
res64: play.api.libs.json.JsValue = {"field4":0,"field5":0,"field6":0}
Again a bit thanks to Travis :-)

Related

Scala parses non-canonical JSON

The following data can be seen with different value types. How can I get the desired output?
package ceshi
import scala.util.parsing.json.JSON
object ceshi1212 {
def main(args: Array[String]): Unit = {
class CC[T] {
def unapply(a: Any): Option[T] = Some(a.asInstanceOf[T])
}
object M extends CC[Map[String, Any]]
object L extends CC[List[Any]]
object S extends CC[String]
val jsonString =
"""
{
"languages": [
{
"name": "English",
"is_active": "true",
"completeness": "asdf"
},
{
"name": "Latin",
"is_active": "asdf",
"completeness": "232"
}
,{
"name": "Latin",
"is_active": "0009"
}
,
"error"
]
}
""".stripMargin
// 不规则json error和并列的数组类型不同 怎么解析自动跳过?
val result = for {
Some(M(map)) <- List(JSON.parseFull(jsonString))
L(languages) = map("languages")
M(language) <- languages
S(name) = language("name")
S(active) = language("is_active")
S(completeness) = language.getOrElse("completeness","--")
} yield {
(name, active, completeness)
}
println(result)
//i want result is: List((English,true,asdf), (Latin,asdf,232),(Latain,0009,""))
}
}
i want get result is List((English,true,asdf), (Latin,asdf,232),(Latain,0009,""))
note: 1 The string is not always at the end of the array, and the position is indeterminate
2 The three keys I need may not be complete
As said in the comments there are other libraries to be recommended for working with json have a look at this post to get an overview: What JSON library to use in Scala?
Answer to your question with specific framework (play-json)
Personally I can recommend to use the play json framework.
To obtain the result you have described with play json, your code might look like this:
import play.api.libs.json._
val json: JsValue = Json.parse(jsonString)
val list = (json \ "languages").as[Seq[JsValue]]
val names = list.map(x => ((x\"name").validate[String] match {
case JsSuccess(v, p ) => v
case _ => ""
}
))
val isActives = list.map(x => ((x\"is_active").validate[String] match {
case JsSuccess(v, p ) => v
case _ => ""
}
))
val completeness = list.map(x => ((x\"completeness").validate[String] match {
case JsSuccess(v, p ) => v
case _ => ""
}
))
// need to know in advance what is your max length of your tuple (tmax)
// since 3rd value "completeness" can be missing, so we just take "" instead
val tmax = 3
val res = for(idx <-0 to tmax-1) yield (names(idx),isActives(idx),completeness(idx))
res.toList
// List[(String, String, String)] = List((English,true,asdf), (Latin,asdf,232), (Latin,0009,""))
There's also a very good documentation for the play json framework, just check it out yourself: https://www.playframework.com/documentation/2.8.x/ScalaJson
If you can switch parser library to circe, you can deal with this types of bad data.
Given you have data model
import io.circe.generic.semiauto._
import io.circe.parser.decode
import io.circe.{Decoder, Json}
case class Languages(languages: Seq[Language])
case class Language(name: String, is_active: String, completeness: Option[String])
You can define a fault-tolerant seq decoder that would skip bad data rather than crash whole parse
def tolerantSeqDecoder[A: Decoder]: Decoder[Seq[A]] = Decoder.decodeSeq(Decoder[A]
.either(Decoder[Json])).map(_.flatMap(_.left.toOption))
and the rest...
val jsonString = """
{
"languages": [
{
"name": "English",
"is_active": "true",
"completeness": "asdf"
},
{
"name": "Latin",
"is_active": "asdf",
"completeness": "232"
},
{
"name": "Latin",
"is_active": "0009"
},
"error"
]
}
"""
val languageDecoder = deriveDecoder[Language]
implicit val tolerantDecoder = tolerantSeqDecoder[Language](languageDecoder)
implicit val languagesDecoder = deriveDecoder[Languages]
val parsed = decode[Languages](jsonString)
println(parsed)
out:
Right(Languages(List(Language(English,true,Some(asdf)), Language(Latin,asdf,Some(232)), Language(Latin,0009,None))))
This approach was suggested by one of circe developers: How do I ignore decoding failures in a JSON array?

Transform all keys from `underscore` to `camel case` of json objects in circe

Origin
{
"first_name" : "foo",
"last_name" : "bar",
"parent" : {
"first_name" : "baz",
"last_name" : "bazz",
}
}
Expected
{
"firstName" : "foo",
"lastName" : "bar",
"parent" : {
"firstName" : "baz",
"lastName" : "bazz",
}
}
How can I transform all keys of json objects ?
Here's how I'd write this. It's not as concise as I'd like, but it's not terrible:
import cats.free.Trampoline
import cats.std.list._
import cats.syntax.traverse._
import io.circe.{ Json, JsonObject }
/**
* Helper method that transforms a single layer.
*/
def transformObjectKeys(obj: JsonObject, f: String => String): JsonObject =
JsonObject.fromIterable(
obj.toList.map {
case (k, v) => f(k) -> v
}
)
def transformKeys(json: Json, f: String => String): Trampoline[Json] =
json.arrayOrObject(
Trampoline.done(json),
_.traverse(j => Trampoline.suspend(transformKeys(j, f))).map(Json.fromValues),
transformObjectKeys(_, f).traverse(obj => Trampoline.suspend(transformKeys(obj, f))).map(Json.fromJsonObject)
)
And then:
import io.circe.literal._
val doc = json"""
{
"first_name" : "foo",
"last_name" : "bar",
"parent" : {
"first_name" : "baz",
"last_name" : "bazz"
}
}
"""
def sc2cc(in: String) = "_([a-z\\d])".r.replaceAllIn(in, _.group(1).toUpperCase)
And finally:
scala> import cats.std.function._
import cats.std.function._
scala> transformKeys(doc, sc2cc).run
res0: io.circe.Json =
{
"firstName" : "foo",
"lastName" : "bar",
"parent" : {
"firstName" : "baz",
"lastName" : "bazz"
}
}
We probably should have some way of recursively applying a Json => F[Json] transformation like this more conveniently.
Depending on your full use-case, with the latest Circe you might prefer just leveraging the existing decoder/encoder for converting between camel/snake according to these references:
https://dzone.com/articles/5-useful-circe-feature-you-may-have-overlooked
https://github.com/circe/circe/issues/663
For instance, in my particular use-case this makes sense because I'm doing other operations that benefit from the type-safety of first deserializing into case classes. So if you're willing to decode the JSON into a case class, and then encode it back into JSON, all you would need is for your (de)serializing code to extend a trait that configures this, like:
import io.circe.derivation._
import io.circe.{Decoder, Encoder, ObjectEncoder, derivation}
import io.circe.generic.auto._
import io.circe.parser.decode
import io.circe.syntax._
trait JsonSnakeParsing {
implicit val myCustomDecoder: Decoder[MyCaseClass] = deriveDecoder[MyCaseClass](io.circe.derivation.renaming.snakeCase)
// only needed if you want to serialize back to snake case json:
// implicit val myCustomEncoder: ObjectEncoder[MyCaseClass] = deriveEncoder[MyCaseClass](io.circe.derivation.renaming.snakeCase)
}
For example, I then extend that when I actually parse or output the JSON:
trait Parsing extends JsonSnakeParsing {
val result: MyCaseClass = decode[MyCaseClass](scala.io.Source.fromResource("my.json").mkString) match {
case Left(jsonError) => throw new Exception(jsonError)
case Right(source) => source
}
val theJson = result.asJson
}
For this example, your case class might look like:
case class MyCaseClass(firstName: String, lastName: String, parent: MyCaseClass)
Here's my full list of circe dependencies for this example:
val circeVersion = "0.10.0-M1"
"io.circe" %% "circe-generic" % circeVersion,
"io.circe" %% "circe-parser" % circeVersion,
"io.circe" %% "circe-generic-extras" % circeVersion,
"io.circe" %% "circe-derivation" % "0.9.0-M5",
def transformKeys(json: Json, f: String => String): TailRec[Json] = {
if(json.isObject) {
val obj = json.asObject.get
val fields = obj.toList.foldLeft(done(List.empty[(String, Json)])) { (r, kv) =>
val (k, v) = kv
for {
fs <- r
fv <- tailcall(transformKeys(v, f))
} yield fs :+ (f(k) -> fv)
}
fields.map(fs => Json.obj(fs: _*))
} else if(json.isArray) {
val arr = json.asArray.get
val vsRec = arr.foldLeft(done(List.empty[Json])) { (vs, v) =>
for {
s <- vs
e <- tailcall(transformKeys(v, f))
} yield s :+ e
}
vsRec.map(vs => Json.arr(vs: _*))
} else {
done(json)
}
}
Currently I do transform like this, but is rather complicated, hope there is a simple way.
I took #Travis answer and modernized it a bit, I took his code and I had several error and warnings, so the updated version for Scala 2.12 with Cats 1.0.0-MF:
import io.circe.literal._
import cats.free.Trampoline, cats.instances.list._, cats.instances.function._, cats.syntax.traverse._, cats.instances.option._
def transformKeys(json: Json, f: String => String): Trampoline[Json] = {
def transformObjectKeys(obj: JsonObject, f: String => String): JsonObject =
JsonObject.fromIterable(
obj.toList.map {
case (k, v) => f(k) -> v
}
)
json.arrayOrObject(
Trampoline.done(json),
_.toList.traverse(j => Trampoline.defer(transformKeys(j, f))).map(Json.fromValues(_)),
transformObjectKeys(_, f).traverse(obj => Trampoline.defer(transformKeys(obj, f))).map(Json.fromJsonObject)
)
}
def sc2cc(in: String) = "_([a-z\\d])".r.replaceAllIn(in, _.group(1).toUpperCase)
def camelizeKeys(json: io.circe.Json) = transformKeys(json, sc2cc).run

In Scala, how to write an efficient json formatter for Map[IndexedSeq[String], Int]?

I think there is not a default format for Map[IndexedSeq[String], Int] in scala (right?) So I've written my own format as follows, however it's very slow. Is there a better way to do this?
class IndexedSeqToIntMapFormat() extends Format[Map[IndexedSeq[String], Int]] {
def writes(o: Map[IndexedSeq[String], Int]): JsValue = {
val mapItems: Seq[String] = o.toSeq.map{case (rowKey, index) => (index.toString +: rowKey).mkString(",")}
Json.obj("items" -> Json.toJson(mapItems))
}
def reads(json: JsValue): JsResult[Map[IndexedSeq[String], Int]] = {
val mapItemsAsString: IndexedSeq[String] = (json \ "items").as[IndexedSeq[String]]
val map: Map[IndexedSeq[String], Int] = mapItemsAsString.map(itemAsString => {
val item: IndexedSeq[String] = itemAsString.split(",").toIndexedSeq
val rowKey: IndexedSeq[String] = item.tail
val rowIndex: Int = item.head.toInt
(rowKey, rowIndex)
}).toMap
JsSuccess(map)
}
}
Thanks!
Can't say for sure whether the following approach is significantly faster than yours, but to my understanding, it does much more conform to the "spirit" of JSON. In a JSON serialization, each object and all its sub-objects and attributes should be named, by what they are. A list of custom string serializations of complex objects is not an actual JSON representation in my opinion.
Doing it in a "proper" JSON way should at least save some time in the parsing, as it does not require additional parsing work on strings but already provides all data in the needed places. And the code looks much more readable :-)
The resulting JSON should look like this:
"items": [
{ "keySeq": [ "key1", "key2", "key3" ], "value": 42 },
{ "keySeq": [ "key4", "key5" ], "value": 123 },
{ "keySeq": [ "key6", "key7", "key7" ], "value": 650 }
]
The formatter could be something like this:
class IndexedSeqToIntMapFormat() extends Format[Map[IndexedSeq[String], Int]] {
def writes(m: Map[IndexedSeq[String], Int]): JsValue = {
val objs = m.toSeq.map { case (keySeq, value) =>
Json.obj("keySeq" -> Json.toJson(keySeq), "value" -> JsNumber(value))
}
Json.obj("items" -> JsArray(objs))
}
def reads(json: JsValue): JsResult[Map[IndexedSeq[String], Int]] = {
val seq = (json \ "items").as[Seq[JsValue]] map { obj =>
( (obj \ "keySeq").as[IndexedSeq[String]], (obj \ "value").as[Int] )
}
JsSuccess(seq.toMap)
}
}
By the way, just out of curiosity - can you tell me in what context you need a such map?

Json.writes[List[Foo]] as "map"

I have a case class Foo(bars: List[Bar]) who is rendered as json via Json inception as an object with an array :
{"bars": [
{
"key: "4587-der",
"value": "something"
}
]
}
But I want to render the bars: List[Bar] as a "map" where Bar.key is used as key :
{"bars":{
"4587-der": {
"value": "something"
}
}
}
How can I obtains that without modifying my case class Foo ?
Thanks a lot
You can define a Writes for Bar by extending Writes[Bar] and implementing a writes method for it:
case class Bar(key: String, value: String)
implicit object BarWrites extends Writes[Bar] {
def writes(bar: Bar): JsValue = Json.obj(
bar.key -> Json.obj("value" -> bar.value)
)
}
scala> Json.stringify(Json.toJson(Bar("4587-der", "something")))
res0: String = {"4587-der":{"value":"something"}}
For those that may be interested, here is a (somewhat) crude implementation of Reads[Bar]:
implicit object BarReads extends Reads[Bar] {
def reads(js: JsValue): JsResult[Bar] = js match {
case JsObject(Seq((key, JsObject(Seq(("value", JsString(value))))))) => JsSuccess(Bar(key, value))
case _ => JsError(Seq())
}
}
scala> Json.parse(""" [{"4587-der":{"value": "something"}}] """).validate[List[Bar]]
res11: play.api.libs.json.JsResult[List[Bar]] = JsSuccess(List(Bar(4587-der,something)),)
Edit, since the OP wants the Bars merged into an object rather than an array:
You'll also have to define a special Writes[List[Bar]] as well:
implicit object BarListWrites extends Writes[List[Bar]] {
def writes(bars: List[Bar]): JsValue =
bars.map(Json.toJson(_).as[JsObject]).foldLeft(JsObject(Nil))(_ ++ _)
}
scala> val list = List(Bar("4587-der", "something"), Bar("1234-abc", "another"))
list: List[Bar] = List(Bar(4587-der,something), Bar(1234-abc,another))
scala> Json.stringify(Json.toJson(list))
res1: String = {"4587-der":{"value":"something"},"1234-abc":{"value":"another"}}

How do I avoid nested Array when serializing a Map with Play Writes?

I am trying to serialize a map using the Json library from Play. I wrote my own Writes since there is none for Maps.
import play.api.libs.json.Json._
import play.api.libs.json._
object NestedArray extends App {
val m: Map[Int, String] = Map(1 -> "one", 2 -> "two")
implicit val mWrites = new Writes[Map[Int, String]] {
def writes(m: Map[Int, String]) = arr(
m.keys.map(k => {
obj("key" -> k.toString,
"value" -> m(k)
)
})
)
}
val j = toJson[Map[Int, String]](m)
println(prettyPrint(j))
}
The output is this:
[ [ {
"key" : "1",
"value" : "one"
}, {
"key" : "2",
"value" : "two"
} ] ]
As you can see there are two pairs of [ ] around the items. When I use a Wrapper class around the map I only get one pair of [ ].
case class Wrap(m: Map[Int, String])
val w = new Wrap(m)
implicit val wrapWrites = new Writes[Wrap] {
def writes(w: Wrap) = obj(
"m" -> w.m.keys.map(k => {
obj("key" -> k.toString,
"value" -> w.m(k)
)
})
)
}
val j2 = toJson[Wrap](w)
println(prettyPrint(j2))
Output:
{
"m" : [ {
"key" : "1",
"value" : "one"
}, {
"key" : "2",
"value" : "two"
} ]
}
Is there a way to achieve that without a wrapper class?
Json.arr makes a JSON array from it's argument list. Since the first argument is itself a sequence, the result is a sequence of a sequence.
E.g.
scala> Json.arr(1,2,3)
res1: play.api.libs.json.JsArray = [1,2,3]
scala> Json.arr(List(1,2,3))
res2: play.api.libs.json.JsArray = [[1,2,3]]
Removing the call to arr and converting the Iterable directly to JSON using toJson removes the nested array
import play.api.libs.json.Json._
import play.api.libs.json._
object NestedArray extends App {
val m: Map[Int, String] = Map(1 -> "one", 2 -> "two")
implicit val mWrites = new Writes[Map[Int, String]] {
def writes(m: Map[Int, String]): JsValue =
Json.toJson(m.keys.map(k => {
obj("key" -> k.toString,
"value" -> m(k)
)
}))
}
val j = toJson[Map[Int, String]](m)
println(prettyPrint(j))
}
Play does provide a Writes[Map[String, _]], which you can probably adapt for your uses. It seems unnecessary to create structure to represent a sequence of key-value pairs, when that's what a JSON object is already. Please see my answer to a similar question: How to serialize a Map[CustomType, String] to JSON