Scala 2.12 here trying to use Lift-JSON to parse a config file. I have the following myapp.json config file:
{
"health" : {
"checkPeriodSeconds" : 10,
"metrics" : {
"stores" : {
"primary" : "INFLUX_DB",
"fallback" : "IN_MEMORY"
}
}
}
}
And the following MyAppConfig class:
case class MyAppConfig()
My myapp.json is going to evolve and potentially become very large with lots of nested JSON structures inside of it. I don't want to have to create Scala objects for each JSON object and then inject that in MyAppConfig like so:
case class Stores(primary : String, fallback : String)
case class Metrics(stores : Stores)
case class Health(checkPeriodSeconds : Int, metrics : Metrics)
case class MyAppConfig(health : Health)
etc. The reason for this is I'll end up with "config object sprawl" with dozens upon dozens of case classes that are only in existence to satisfy serialization from JSON into Scala-land.
Instead, I'd like to use Lift-JSON to read the myapp.json config file, and then have MyAppConfig just have helper functions that read/parse values out of the JSON on the fly:
import net.liftweb.json._
// Assume we instantiate MyAppConfig like so:
//
// val json = Source.fromFile(configFilePath)
// val myAppConfig : MyAppConfig = new MyAppConfig(json.mkString)
//
class MyAppConfig(json : String) {
implicit val formats = DefaultFormats
def primaryMetricsStore() : String = {
// Parse "INFLUX_DB" value from health.metrics.stores.primary
}
def checkPeriodSeconds() : Int = {
// Parse 10 value from health.checkPeriodSeconds
}
}
This way I can cherry pick which configs I want to expose (make readable) to my application. I'm just not following the Lift API docs to see how this strategy is possible, they all seem to want me to go with creating tons of case classes. Any ideas?
Case classes are not mandatory for extracting data from JSON. You can query the parsed tree and transfrom data according to your needs. The values from the example can be extracted as follows:
import net.liftweb.json._
class MyAppConfig(json : String) {
private implicit val formats = DefaultFormats
private val parsed = parse(json)
def primaryMetricsStore() : String = {
(parsed \ "health" \ "metrics" \ "stores" \ "primary").extract[String]
}
def checkPeriodSeconds() : Int = {
(parsed \ "health" \ "checkPeriodSeconds").extract[Int]
}
}
The original doc provides all the details.
Related
I started with the accepted answer to SO question 53573659 which has a nested list of attrs and uses the auto-parser to get the data into case classes. I want to be able to handle the same data but with the nested fields having kebab-case rather than camel case.
Here is the same input JSON with the kebab-case fields
val sampleKebab="""{
"parent" : {
"name" : "title",
"items" : [
{
"foo" : "foo1",
"attrs" : {
"attr-a" : "attrA1",
"attr-b" : "attrB1"
}
},
{
"foo" : "foo2",
"attrs" : {
"attr-a" : "attrA2",
"attr-b" : "attrB2",
"attr-c" : "attrC2"
}
}
]
}
}"""
I can decode the attrs data by itself using the following example
import io.circe.derivation.deriveDecoder
import io.circe.{Decoder, derivation}
import io.circe.generic.auto._
import io.circe.parser._
val attrKebabExample = """{
"attr-a": "attrA2",
"attr-b": "attrB2",
"attr-c": "attrC2"
}"""
case class AttrsKebab(attrA: String, attrB: String)
implicit val decoder: Decoder[AttrsKebab] = deriveDecoder(derivation.renaming.kebabCase)
val attrKebabData = decode[AttrsKebab](attrKebabExample)
attrKebabData decodes to
Either[io.circe.Error,AttrsKebab] = Right(AttrsKebab(attrA2,attrB2))
When I try to tie this decoder into the case class hierarchy from the original question, it exposes some glue that I am missing to hold it all together
case class ItemKebab(foo: String, attrs : AttrsKebab)
case class ParentKebab(name: String, items: List[ItemKebab])
case class DataKebab(parent : ParentKebab)
case class Data(parent : Parent)
val dataKebab=decode[DataKebab](sample)
In this case, dataKebab contains a DecodingFailure
Either[io.circe.Error,DataKebab] = Left(DecodingFailure(Attempt to decode value on failed cursor, List(DownField(attr-a), DownField(attrs), DownArray, DownField(items), DownField(parent))))
My guess is that either the decoder I defined is being ignored, or I need to explicitly define more of the decode process, but I'm looking for some help to find what the solution might be.
So I have a json string in scala which looks something like this
"""
{
"input": {
"House" :{
"Tile" : "Ceramic"
"Kitchen" : {
"Sink" : "Stainless-Steel"
"Counter-Top" : "Granite"
}
}
}
}
"""
I'm using json4s to parse this and to put it into a Map and then I'm making it mutable so I can edit it, but I don't really know how to take it and modify it to make it a new json object.
import org.json4s.jackson.JsonMethods._
import org.json4s._
import org.json4s.native.Serialization._
import org.json4s.native.Serialization
import scala.collection.mutable
...
val reqJsonMap = parse(reqJson).extract[Map[String,Any]]
val reqJsonMutableMap= collection.mutable.Map[String,Any]()
reqJsonMutableMap ++=reqJsonMap
What I want to do is I want to edit it and make the json of Tile=Marble and change the key of Kitchen to Bathroom
I just don't know how to turn that object into this
"""
{
"input": {
"House" :{
"Tile" : "Marble"
"Bathroom" : {
"Sink" : "Stainless-Steel"
"Counter-Top" : "Granite"
}
}
}
}
"""
I think it is far better to work with case classes than with JSON. I think it is the most convenient to use a Json parsing library such as circe to parse your Json string into a case class, and then do whatever thing you need to do with it:
import io.circe._, io.circe.generic.auto._, io.circe.syntax._, io.circe.parser._
val myJsonString = """
{
"input": {
"house" :{
"tile" : "Ceramic",
"kitchen" : {
"sink" : "Stainless-Steel",
"counterTop" : "Granite"
}
}
}
}
"""
case class Data(input: Input)
case class Input(house: HouseInfo)
case class HouseInfo(tile: String, kitchen: KitchenInfo)
case class KitchenInfo(sink: String, counterTop: String)
val maybeDataJsonObject = parse(myJsonString)
val maybeData = maybeDataJsonObject match {
case Right(dataJsonObject) => dataJsonObject.as[Data]
case Left(failure) => println(failure)
}
maybeData match {
case Right(data) => println(data)
case Left(failure) => println(failure)
}
By the way, the format of your json is not good, because you have capitalized the initials of the field names, which should conventionally be lower cased. Also, you have written it as Counter-Top, instead of let’s say counterTop and the - character would be problematic in field names in Scala.
For applying the transformations you are looking for, you need to write new case classes with desired field names, then you can write a transformer function to transform the obtained Data object into the TransformedData one. Then you would be able to use circe to obtain a new Json string by coding the case class in Json, like in this:
val transformedJsonString = myTransformedData.asJson.toString
I have many very large json-objects that I return from Play Framework with Scala.
In most cases the user doesn't need all the data in the objects, only a few fields. So I want to pass in the paths I need (as query parameters), and return a subset of the json object.
I have looked at using JSON Transformers for this task.
Filter code
def filterByPaths(paths: List[JsPath], inputObject: JsObject) : JsObject = {
paths
.map(_.json.pick)
.map(inputObject.transform)
.filter(_.isSuccess)
.map { case JsSuccess(value, path) => (value, path) }
.foldLeft(Json.obj()) { (obj, jsValueAndPath) =>
val(jsValue, path) = jsValueAndPath
val transformer = __.json.update(path.json.put(jsValue))
obj.transform(transformer).get
}
}
Usage:
val input = Json.obj(
"field1" -> Json.obj(
"field2" -> "right result"
),
"field4" -> Json.obj(
"field5" -> "not included"
),
)
val result = filterByPaths(List(JsPath \ "field1" \ "field2"), input)
// {"field1":{"field2":"right result"}}
Problem
This code works fine for JsObjects. But I can't make it work if there are JsArrays in the strucure. I had hoped that my JsPath could contain an index to look up the field, but that's not the case. (Don't know why I assumed that, maybe my head was too far in the JavaScript-world)
So this would fail to return the first entry in the Array:
val input: JsObject = Json.parse("""
{
"arr1" : [{
"field1" : "value1"
}]
}
""").as[JsObject]
val result = filterByPaths(List(JsPath \ "arr1" \ "0"), input)
// {}
Question
My question is: How can I return a subset of a json structure that contains arrays?
Alternative solution
I have the data as a case class first, and I serialize it to Json, and then run filterByPaths on it. Having a Reader that only creates the json I need in the first place might be a better solution, but creating a Reader on the fly, with configuration from queryparams seamed a more difficult task, then just stripping down the json afterwards.
The example of the returning array element:
val input: JsValue = Json.parse("""
{
"arr1" : [{
"field1" : "value1"
}]
}
""")
val firstElement = (input \ "arr1" \ 0).get
val firstElementAnotherWay = input("arr1")(0)
More about this in the Play Framework documentation: https://www.playframework.com/documentation/2.6.x/ScalaJson
Update
It looks like you got the old issue RuntimeException: expected KeyPathNode. JsPath.json.put, JsPath.json.update can't past an object to a nesting array.
https://github.com/playframework/playframework/issues/943
https://github.com/playframework/play-json/issues/82
What you can do:
Use the JSZipper: https://github.com/mandubian/play-json-zipper
Create a script to update arrays "manually"
If you can afford it, strip array in a resulting object
Example of stripping array (point 3):
def filterByPaths(paths: List[JsPath], inputObject: JsObject) : JsObject = {
paths
.map(_.json.pick)
.map(inputObject.transform)
.filter(_.isSuccess)
.map { case JsSuccess(value, path) => (value, path)}
.foldLeft(Json.obj()) { (obj, jsValueAndPath) =>
val (jsValue, path) = jsValueAndPath
val arrayStrippedPath = JsPath(path.path.filter(n => !(n.toJsonString matches """\[\d+\]""")))
val transformer = __.json.update(arrayStrippedPath.json.put(jsValue))
obj.transform(transformer).get
}
}
val result = filterByPaths(List(JsPath \ "arr1" \ "0"), input)
// {"arr1":{"field1":"value1"}}
The example
The best to handle JSON objects is by using case classes and create implicit Reads and Writes, by that you can handle errors every fields directly. Don't make it complicated.
Don't use .get() much recommended to use .getOrElse() because scala is a type-safe programming language.
Don't just use any Libraries except you know the process behind it, much better to create your own parsing method with simplified solution to save memory.
I hope it will help you..
Scala 2.12 here. I'm trying to use Lift-JSON to deserialize some JSON into a Scala object and am having trouble navigating the Lift API. Please note: I'm not married to Lift-JSON, any other working solution will be accepted so long as I don't have to bring any heavy/core Play dependencies into my project.
Here's the JSON file I'm trying to read:
{
"fizz" : "buzz",
"foo" : [
"123",
"456",
"789"
],
"bar" : {
"whistle" : 1,
"feather" : true
}
}
Here's my Scala object hierarchy:
case class Bar(whistle : Integer, feather : Boolean)
case class MyConfig(fizz : String, foo : Array[String], bar : Bar)
And finally my best attempt at the codeup for this:
def loadConfig(configFilePath : String) : MyConfig = {
val configJson = Source.fromFile(configFilePath)
val parsedJson = parse(configJson.mkString)
MyConfig(???)
}
I need validation in place so that if the JSON is not valid an exception is thrown. Any ideas how I can extract fields out of parsedJson and use them to set values for my MyConfig instance? And how to perform the validation?
Have you tried parsedJson.extract[MyConfig]? That is straight out of the Extracting values documentation. If you haven't already, you will need to specify an implicit reference to the default formats:
implicit val formats = DefaultFormats
I want to have different names of fields in my case classes and in my JSON, therefore I need a comfortable way of renaming in both, encoding and decoding.
Does someone have a good solution ?
You can use Custom key mappings via annotations. The most generic way is the JsonKey annotation from io.circe.generic.extras._. Example from the docs:
import io.circe.generic.extras._, io.circe.syntax._
implicit val config: Configuration = Configuration.default
#ConfiguredJsonCodec case class Bar(#JsonKey("my-int") i: Int, s: String)
Bar(13, "Qux").asJson
// res5: io.circe.Json = JObject(object[my-int -> 13,s -> "Qux"])
This requires the package circe-generic-extras.
Here's a code sample for Decoder (bit verbose since it won't remove the old field):
val pimpedDecoder = deriveDecoder[PimpClass].prepare {
_.withFocus {
_.mapObject { x =>
val value = x("old-field")
value.map(x.add("new-field", _)).getOrElse(x)
}
}
}
implicit val decodeFieldType: Decoder[FieldType] =
Decoder.forProduct5("nth", "isVLEncoded", "isSerialized", "isSigningField", "type")
(FieldType.apply)
This is a simple way if you have lots of different field names.
https://circe.github.io/circe/codecs/custom-codecs.html
You can use the mapJson function on Encoder to derive an encoder from the generic one and remap your field name.
And you can use the prepare function on Decoder to transform the JSON passed to a generic Decoder.
You could also write both from scratch, but it may be a ton of boilerplate, those solutions should both be a handful of lines max each.
The following function can be used to rename a circe's JSON field:
import io.circe._
object CirceUtil {
def renameField(json: Json, fieldToRename: String, newName: String): Json =
(for {
value <- json.hcursor.downField(fieldToRename).focus
newJson <- json.mapObject(_.add(newName, value)).hcursor.downField(fieldToRename).delete.top
} yield newJson).getOrElse(json)
}
You can use it in an Encoder like so:
implicit val circeEncoder: Encoder[YourCaseClass] = deriveEncoder[YourCaseClass].mapJson(
CirceUtil.renameField(_, "old_field_name", "new_field_name")
)
Extra
Unit tests
import io.circe.parser._
import org.specs2.mutable.Specification
class CirceUtilSpec extends Specification {
"CirceUtil" should {
"renameField" should {
"correctly rename field" in {
val json = parse("""{ "oldFieldName": 1 }""").toOption.get
val resultJson = CirceUtil.renameField(json, "oldFieldName", "newFieldName")
resultJson.hcursor.downField("oldFieldName").focus must beNone
resultJson.hcursor.downField("newFieldName").focus must beSome
}
"return unchanged json if field is not found" in {
val json = parse("""{ "oldFieldName": 1 }""").toOption.get
val resultJson = CirceUtil.renameField(json, "nonExistentField", "newFieldName")
resultJson must be equalTo json
}
}
}
}