Parsing JSON with Scala play framework - handling variable array elements? - json

I'm trying to parse a json string of search results of the following form:
"""
{
"metadata": [
"blah",
"unimportant"
],
"result": [
{
"type": "one",
"title": "this",
"weird": "attribute unique to this type",
"other": 7
},
{
"type": "two",
"title": "that",
"another_weird": "a different attribute unique to this second type",
"another": "you get the idea"
},
{
"type": "one",
"title": "back to this type, which has the same fields as the other element of this type",
"weird": "yep",
"other": 8
}
...
]
}
"""
There is a known, fixed number of result element types (given by the type field), each of which correspond to a unique schema. For a given search request, there can be any number of each type returned in any order (up to some fixed total).
Writing out the case classes and Reads implicits for each type is easy enough, but my question is around the best way to handle the variability in the result sequence... pattern matching seems the obvious choice, but I'm just not sure where or how with the play framework. Thanks in advance!
EDIT: Per the suggestion in the comments, I gave it a go by attempting to read the result sequence as subtypes of a common base trait, but that didn't quite work. The following compiles, but doesn't parse the example json correctly. Any additional suggestions are welcome.
sealed trait Parent {def title: String}
case class One(`type`: String, title: String, weird: String, other: Int) extends Parent
case class Two(`type`: String, title: String, another_weird: String, another: String) extends Parent
case class SearchResponse(result: Seq[Parent], metadata: Seq[String])
implicit val oneReader = Json.reads[One]
implicit val twoReader = Json.reads[Two]
implicit val parentReader = Json.reads[Parent]
implicit val searchReader = (
(__ \ "result").read[Seq[Parent]] and (__ \ "metadata").read[Seq[String]]
)(SearchResponse.apply _)
val searchResult = Json.fromJson[SearchResponse](json)
print(searchResult)

Define implicit JsonConfiguration
import play.api.libs.json._
sealed trait Parent {
def title: String
}
case class One(title: String, weird: String, other: Int) extends Parent
case class Two(title: String, another_weird: String, another: String) extends Parent
case class SearchResponse(result: Seq[Parent], metadata: Seq[String])
implicit val cfg = JsonConfiguration(
discriminator = "type",
typeNaming = JsonNaming(_.toLowerCase)
)
implicit val oneReader = Json.reads[One]
implicit val twoReader = Json.reads[Two]
implicit val parentReader = Json.reads[Parent]
implicit val searchReader = Json.reads[SearchResponse]
val searchResult = Json.fromJson[SearchResponse](json)
println(searchResult)
// JsSuccess(SearchResponse(List(One(this,attribute unique to this type,7), Two(that,a different attribute unique to this second type,you get the idea), One(back to this type, which has the same fields as the other element of this type,yep,8)),List(blah, unimportant)),)
https://www.playframework.com/documentation/2.8.x/ScalaJsonAutomated#Custom-Naming-Strategies

Related

Circe Couldn't convert raw json to case class Error: could not find Lazy implicit value of type io.circe.generic.decoding.DerivedDecoder

I have defined a couple of case classes for JSON representation but I am not sure whether I did it properly as there a lot of nested case classes.
Entities like spec, meta and so on are of type JSONObject as well as the Custom object itself.
Here is all the classes I have defined:
case class CustomObject(apiVersion: String,kind: String, metadata: Metadata,spec: Spec,labels: Object,version: String)
case class Metadata(creationTimestamp: String, generation: Int, uid: String,resourceVersion: String,name: String,namespace: String,selfLink: String)
case class Spec(mode: String,image: String,imagePullPolicy: String, mainApplicationFile: String,mainClass: String,deps: Deps,driver: Driver,executor: Executor,subresources: Subresources)
case class Driver(cores: Double,coreLimit: String,memory: String,serviceAccount: String,labels: Labels)
case class Executor(cores: Double,instances: Double,memory: String,labels: Labels)
case class Labels(version: String)
case class Subresources(status: Status)
case class Status()
case class Deps()
And this is a JSON structure for the custom K8s object I need to transform:
{
"apiVersion": "sparkoperator.k8s.io/v1alpha1",
"kind": "SparkApplication",
"metadata": {
"creationTimestamp": "2019-01-11T15:58:45Z",
"generation": 1,
"name": "spark-example",
"namespace": "default",
"resourceVersion": "268972",
"selfLink": "/apis/sparkoperator.k8s.io/v1alpha1/namespaces/default/sparkapplications/spark-example",
"uid": "uid"
},
"spec": {
"deps": {},
"driver": {
"coreLimit": "1000m",
"cores": 0.1,
"labels": {
"version": "2.4.0"
},
"memory": "1024m",
"serviceAccount": "default"
},
"executor": {
"cores": 1,
"instances": 1,
"labels": {
"version": "2.4.0"
},
"memory": "1024m"
},
"image": "gcr.io/ynli-k8s/spark:v2.4.0,
"imagePullPolicy": "Always",
"mainApplicationFile": "http://localhost:8089/spark_k8s_airflow.jar",
"mainClass": "org.apache.spark.examples.SparkExample",
"mode": "cluster",
"subresources": {
"status": {}
},
"type": "Scala"
}
}
UPDATE:
I want to convert JSON into case classes with Circe, however, with such classes I face this error:
Error: could not find Lazy implicit value of type io.circe.generic.decoding.DerivedDecoder[dataModel.CustomObject]
implicit val customObjectDecoder: Decoder[CustomObject] = deriveDecoder[CustomObject]
I have defined implicit decoders for all case classes:
implicit val customObjectLabelsDecoder: Decoder[Labels] = deriveDecoder[Labels]
implicit val customObjectSubresourcesDecoder: Decoder[Subresources] = deriveDecoder[Subresources]
implicit val customObjectDepsDecoder: Decoder[Deps] = deriveDecoder[Deps]
implicit val customObjectStatusDecoder: Decoder[Status] = deriveDecoder[Status]
implicit val customObjectExecutorDecoder: Decoder[Executor] = deriveDecoder[Executor]
implicit val customObjectDriverDecoder: Decoder[Driver] = deriveDecoder[Driver]
implicit val customObjectSpecDecoder: Decoder[Spec] = deriveDecoder[Spec]
implicit val customObjectMetadataDecoder: Decoder[Metadata] = deriveDecoder[Metadata]
implicit val customObjectDecoder: Decoder[CustomObject] = deriveDecoder[CustomObject]
The reason you can't derive a decode for CustomObject is because of the labels: Object member.
In circe all decoding is driven by static types, and circe does not provide encoders or decoders for types like Object or Any, which have no useful static information.
If you change that case class to something like the following:
case class CustomObject(apiVersion: String, kind: String, metadata: Metadata, spec: Spec)
…and leave the rest of your code as is, with the import:
import io.circe.Decoder, io.circe.generic.semiauto.deriveDecoder
And define your JSON document as doc (after adding a quotation mark to the "image": "gcr.io/ynli-k8s/spark:v2.4.0, line to make it valid JSON), the following should work just fine:
scala> io.circe.jawn.decode[CustomObject](doc)
res0: Either[io.circe.Error,CustomObject] = Right(CustomObject(sparkoperator.k8s.io/v1alpha1,SparkApplication,Metadata(2019-01-11T15:58:45Z,1,uid,268972,spark-example,default,/apis/sparkoperator.k8s.io/v1alpha1/namespaces/default/sparkapplications/spark-example),Spec(cluster,gcr.io/ynli-k8s/spark:v2.4.0,Always,http://localhost:8089/spark_k8s_airflow.jar,org.apache.spark.examples.SparkExample,Deps(),Driver(0.1,1000m,1024m,default,Labels(2.4.0)),Executor(1.0,1.0,1024m,Labels(2.4.0)),Subresources(Status()))))
Despite what one of the other answers says, circe can definitely derive encoders and decoders for case classes with no members—that's definitely not the problem here.
As a side note, I wish it were possible to have better error messages than this:
Error: could not find Lazy implicit value of type io.circe.generic.decoding.DerivedDecoder[dataModel.CustomObject
But given the way circe-generic has to use Shapeless's Lazy right now, this is the best we can get. You can try circe-derivation for a mostly drop-in alternative for circe-generic's semi-automatic derivation that has better error messages (and some other advantages), or you can use a compiler plugin like splain that's specifically designed to give better error messages even in the presence of things like shapeless.Lazy.
As one final note, you can clean up your semi-automatic definitions a bit by letting the type parameter on deriveDecoder be inferred:
implicit val customObjectLabelsDecoder: Decoder[Labels] = deriveDecoder
This is entirely a matter of taste, but I find it a little less noisy to read.

Encoding Scala None to JSON value using circe

Suppose I have the following case classes that need to be serialized as JSON objects using circe:
#JsonCodec
case class A(a1: String, a2: Option[String])
#JsonCodec
case class B(b1: Option[A], b2: Option[A], b3: Int)
Now I need to encode val b = B(None, Some(A("a", Some("aa")), 5) as JSON but I want to be able to control whether it is output as
{
"b1": null,
"b2": {
"a1": "a",
"a2": "aa"
},
"b3": 5
}
or
{
"b2": {
"a1": "a",
"a2": "aa"
},
"b3": 5
}
Using Printer's dropNullKeys config, e.g. b.asJson.noSpaces.copy(dropNullKeys = true) would result in omitting Nones from output whereas setting it to false would encode Nones as null (see also this question). But how can one control this setting on a per field basis?
The best way to do this is probably just to add a post-processing step to a semi-automatically derived encoder for B:
import io.circe.{ Decoder, JsonObject, ObjectEncoder }
import io.circe.generic.JsonCodec
import io.circe.generic.semiauto.{ deriveDecoder, deriveEncoder }
#JsonCodec
case class A(a1: String, a2: Option[String])
case class B(b1: Option[A], b2: Option[A], b3: Int)
object B {
implicit val decodeB: Decoder[B] = deriveDecoder[B]
implicit val encodeB: ObjectEncoder[B] = deriveEncoder[B].mapJsonObject(
_.filter {
case ("b1", value) => !value.isNull
case _ => true
}
)
}
And then:
scala> import io.circe.syntax._
import io.circe.syntax._
scala> B(None, None, 1).asJson.noSpaces
res0: String = {"b2":null,"b3":1}
You can adjust the argument to the filter to remove whichever null-valued fields you want from the JSON object (here I'm just removing b1 in B).
It's worth noting that currently you can't combine the #JsonCodec annotation and an explicitly defined instance in the companion object. This isn't an inherent limitation of the annotation—we could check the companion object for "overriding" instances during the macro expansion, but doing so would make the implementation substantially more complicated (right now it's quite simple). The workaround is pretty simple (just use deriveDecoder explicitly), but of course we'd be happy to consider an issue requesting support for mixing and matching #JsonCodec and explicit instances.
Circe have added a method dropNullValues on Json that uses what Travis Brown mentioned above.
def dropNulls[A](encoder: Encoder[A]): Encoder[A] =
encoder.mapJson(_.dropNullValues)
implicit val entityEncoder: Encoder[Entity] = dropNulls(deriveEncoder)

Parse a json array of object to their appropriate case class

I have a json array of settings like so:
[
{
"name": "Company Name",
"key": "company_name",
"default": "Foo"
}, {
"name": "Deposit Weeks",
"key": "deposit_weeks",
"default": 6
}, {
"name": "Is VAT registered",
"key": "vat_registered",
"default": false
}
]
I want to parse this into a Seq of Setting objects. I have tried to define my json format by using a trait and defining the different case classes according to the data type in the json object:
sealed trait Setting
case class StringSetting(name: String, key: String, default: String) extends Setting
case class IntSetting(name: String, key: String, default: Int) extends Setting
case class BoolSetting(name: String, key: String, default: Boolean) extends Setting
Now I try to parse the json:
val json = Json.parse(jsonStr)
implicit val jsonFormat: Format[Setting] = Json.format[Setting]
val result = Try(json.as[Seq[Setting]])
Here I get a compile error:
Error:(19, 61) No unapply or unapplySeq function found
implicit val jsonFormat: Format[Setting] = Json.format[Setting]
Is there a way to map each setting to its appropriate case class?
The naive approach would be to provide Reads[Setting](if your aim just to convert json to object) so that JSON deserializer able to build the right variant of Setting.
import play.api.libs.json._
import play.api.libs.functional.syntax._
implicit val settingReads: Reads[Setting] = (__ \ "default").read[String].map[Setting](StringSetting) |
(__ \ "default").read[Int].map[Setting](IntSetting) |
(__ \ "default").read[Boolean].map[Setting](BoolSetting)
However, this would not work if you have same type for 'default' in different sub classes. In this case JSON deserializer unable to distinguish
between those two case classes.
Another approach is to use play json variant library.
import julienrf.variants.Variants
sealed trait Setting
case class StringSetting(name: String, key: String, default: String) extends Setting
case class IntSetting(name: String, key: String, default: Int) extends Setting
case class BoolSetting(name: String, key: String, default: Boolean) extends Setting
object Setting {
implicit val format: Format[Setting] = Variants.format[Setting]
}
Variant.format provides both read and writes for Setting. Make sure that assignment of 'implicit val format' should happen after all possible subclass has been declared.
For more information regarding play json variant library click here

Decode single field of object in a json array with argonaut/circe

Suppose I have such json
{
"sha": "some sha",
"parents": [{
"url": "some url",
"sha": "some parent sha"
}]
}
and such case class
case class Commit(sha: String, parentShas: List[String])
In play-json I could write the reads like this:
val commitReads: Reads[Commit] = (
(JsPath \ "sha").read[String] and
(JsPath \ "parents" \\ "sha").read[List[String]]
)(Commit.apply _)
I'm looking for a equivalent way of decoding only the "sha" of "parent" in argonaut/circe but I haven't found any. "HCursor/ACursor" has downArray but from there on I don't know what to do. Thank you very much in advance!
Neither circe nor Argonaut keeps track of which fields have been read in JSON objects, so you can just ignore the extra "url" field (just as in Play). The trickier part is finding the equivalent of Play's \\, which circe doesn't have at the moment, although you've convinced me we need to add it.
First of all, this is relatively easy if you have a separate SHA type:
import io.circe.Decoder
val doc = """
{
"sha": "some sha",
"parents": [{
"url": "some url",
"sha": "some parent sha"
}]
}
"""
case class Sha(value: String)
object Sha {
implicit val decodeSha: Decoder[Sha] = Decoder.instance(_.get[String]("sha")).map(Sha(_))
}
case class Commit(sha: Sha, parentShas: List[Sha])
object Commit {
implicit val decodeCommit: Decoder[Commit] = for {
sha <- Decoder[Sha]
parents <- Decoder.instance(_.get[List[Sha]]("parents"))
} yield Commit(sha, parents)
}
Or, using Cats's applicative syntax:
import cats.syntax.cartesian._
implicit val decodeCommit: Decoder[Commit] =
(Decoder[Sha] |#| Decoder.instance(_.get[List[Sha]]("parents"))).map(Commit(_, _))
And then:
scala> import io.circe.jawn._
import io.circe.jawn._
scala> decode[Commit](doc)
res0: cats.data.Xor[io.circe.Error,Commit] = Right(Commit(Sha(some sha),List(Sha(some parent sha))))
But that's not really an answer, since I'm not going to ask you to change your model. :) The actual answer is a bit less fun:
case class Commit(sha: String, parentShas: List[String])
object Commit {
val extractSha: Decoder[String] = Decoder.instance(_.get[String]("sha"))
implicit val decodeCommit: Decoder[Commit] = for {
sha <- extractSha
parents <- Decoder.instance(c =>
c.get("parents")(Decoder.decodeCanBuildFrom[String, List](extractSha, implicitly))
)
} yield Commit(sha, parents)
}
This is bad, and I'm ashamed it's necessary, but it works. I've just filed an issue to make sure this gets better in a future circe release.

json4s parse json partially

I have a json model, where contents of certain attribute depend on the other attribute. Something like this:
"paymentMethod": "CREDIT_CARD",
"metaData": {
"cardType": "VISA",
"panPrefix": "",
"panSuffix": "",
"cardHolder": "",
"expiryDate": ""
}
So when paymentMethod equals to CREDIT_CARD, the metadata object will contain attributes as described. In case of other payment method, there'll be different metadata.
I want to handle this situation in a future-proof way. What I'm trying to do is to not parse the metadata field right away, but keep it somehow "unparsed" until I've parsed the paymentMethod field. Then I'd take the metadata and applied appropriate parsing approach.
However I don't know which type to use for a Scala class field for such "late parsed" attributes. I've tried String, JsonInput, JObject, and they all are not suitable (either don't compile or can't be parsed). Any ideas which type can I use? Or, in other words:
case class CreditCardMetadata(
cardType: String,
panPrefix: String,
panSuffix: String,
cardHolder: String,
expiryDate: String)
case class PaypalMetadata(...) // etc.
case class PaymentGatewayResponse(
paymentMethod: String,
metadata: ???)
You could create a CustomSerializer to parse the metadata directly. Something like :
case class PaymentResponse(payment: Payment, otherField: String)
sealed trait Payment
case class CreditCardPayment(cardType: String, expiryDate: String) extends Payment
case class PayPalPayment(email: String) extends Payment
object PaymentResponseSerializer extends CustomSerializer[PaymentResponse]( format => (
{
case JObject(List(
JField("paymentMethod", JString(method)),
JField("metaData", metadata),
JField("otherField", JString(otherField))
)) =>
implicit val formats = DefaultFormats
val payment = method match {
case "CREDIT_CARD" => metadata.extract[CreditCardPayment]
case "PAYPAL" => metadata.extract[PayPalPayment]
}
PaymentResponse(payment, otherField)
},
{ case _ => throw new UnsupportedOperationException } // no serialization to json
))
Which can be used as:
implicit val formats = DefaultFormats + PaymentResponseSerializer
val json = parse("""
{
"paymentMethod": "CREDIT_CARD",
"metaData": {
"cardType": "VISA",
"expiryDate": "2015"
},
"otherField": "hello"
}
""")
val json2 = parse("""
{
"paymentMethod": "PAYPAL",
"metaData": {
"email": "foo#bar.com"
},
"otherField": "world"
}
""")
val cc = json.extract[PaymentResponse]
// PaymentResponse(CreditCardPayment(VISA,2015),hello)
val pp = json2.extract[PaymentResponse]
// PaymentResponse(PayPalPayment(foo#bar.com),world)
You can use a Map[String, String].
It will contain anything you may need.
The answer by Peter Neyens has inspired me to implement my own solution. It's not as generic as his, but in my case I needed something really simple and ad-hoc. Here's what I've done:
It's possible to define a case class with the field of unknown type is represented by a JObject type. Something like this:
case class PaymentGatewayResponse(
default: Boolean,
paymentMethod: String,
visibleForCustomer: Boolean,
active: Boolean,
metaData: JObject)
When such json is parsed into such case class, this field is not parsed immediately, but contains all the necessary information. Then it's possible parse it in a separate step:
case class CreditCardMetadata(
cardType: String,
cardObfuscatedNumber: String,
cardHolder: String,
expiryDate: String)
val response: PaymentGatewayResponse = doRequest(...)
response.map { r =>
r.paymentMethod match {
case "CREDIT_CARD" => r.metaData.extract[CreditCardMetadata]
case unsupportedType: String => throw new UnsupportedPaymentMethodException(unsupportedType)
}
}