Consider this JSON:
{
"myDocument": {
"static_key": "value",
"dynamic_key": "value",
"static_key2": "value2",
"dynamic_key2": {
"dynamic_key3": "value3"
}
}
}
The JSON documents that I'm going to process have some static keys (fields that I know that are going to be present always) but has some others that could be or not be present, with names unbeknownst in advance for mapping them to some case class.
I have the absolute path to the field in a String (retrieved from a configuration stored in a Database) with the following structure:
"myDocument.dynamic_key2.dynamic_key3"
I know that I need to have an ADT for mapping back and forth, so I came to this:
sealed trait Data
final case class StringTuple(key: String, value: String) extends Data
object StringTuple {
implicit val encoder: Encoder[StringTuple] = deriveEncoder[StringTuple]
implicit val decoder: Decoder[StringTuple] = deriveDecoder[StringTuple]
}
final case class NumericTuple(key: String, value: Double) extends Data
object NumericTuple {
implicit val encoder: Encoder[NumericTuple] = deriveEncoder[NumericTuple]
implicit val decoder: Decoder[NumericTuple] = deriveDecoder[NumericTuple]
}
final case class DateTuple(key: String, value: OffsetDateTime) extends Data
object DateTuple {
implicit val encoder: Encoder[DateTuple] = deriveEncoder[DateTuple]
implicit val decoder: Decoder[DateTuple] = deriveDecoder[DateTuple]
}
final case class TransformedJson(data: Data*)
object TransformedJson {
def apply(data: Data*): TransformedJson = new TransformedJson(data: _*)
implicit val encoder: Encoder[TransformedJson] = deriveEncoder[TransformedJson]
implicit val decoder: Decoder[TransformedJson] = deriveDecoder[TransformedJson]
}
Based on this discussion, it makes no sense to use Map[String, Any] with circe, so I divided the three possible key-value cases that I will encounter when parsing individual fields:
A numeric field, that I'm going to parse as Double.
A String field parsed as is (String).
A date field parsed as OffsetDateTime.
For that reason, I've created three case classes that model these combinations (NumericTuple, StringTuple and DateTuple) and my idea is to produce an output JSON like this:
{
"dynamic_key": "extractedValue",
"dynamic_key3": "extractedValue3",
...
}
("Plain", with no nesting at all).
My idea is to create a list of Data objects to achieve this and I have something like this:
def extractValue(confElement: Conf, json: String) = {
val cursor: HCursor = parse(json).getOrElse(Json.Null).hcursor
val decodeDynamicParam = Decoder[NumericTuple].prepare(
/*
Here I think (not sure) that I can extract the value with the decoder,
but, how can I extract the key name and set it, alongside with the extracted
value?
*/
_.downField(confElement.path)
)
}
Some considerations:
Based on Travis' response to this question I'm trying to model the JSON as closely as possible for working with circe. That's why I tried with the tuples model.
Based (again) on Travis's response to this SO question is that I'm trying with Decode.prepare(...) method. And here comes my question...
Question: How do I extract the specific key name of the current cursor position and map it to the Tuple?
I need only the current key, not all the key set that .keys method of ACursor returns. With that key, I want to map manually the Tuple with the current key name and the extracted value.
For summing it up, my need is to transform a structure that has some unknown keys (name and position), extract their values based on the absolute-dot-separated path that I have and lift both the key name and the value name to a case class that I suffixed as Tuple.
Can you shed some light about this?
Thanks
Related
How to configure the spray-json parsing on parsing options?
Similarly as Jackson Parsing Features.
For example, I am parsing a json that has a field that my case class has not, and it is breaking:
spray.json.DeserializationException: Object is missing required member 'myfield'
UPDATE :
A simple example:
case class MyClass(a: String, b: Long);
and try to parse an incomplete json like
val data = "{a: \"hi\"}"
with a spray-json format like:
jsonFormat2(MyClass.apply)
// ...
data.parseJson.convertTo[MyClass]
(simplified code).
But the question goes further, I want to ask about configuration options like in other parsers. More examples:
Be able to ignore fields that exist in the JSON but not in the case class.
Ways of managing nulls or nonexistent values.
etc.
SprayJson allows you to define custom parsers like so:
case class Foo(a: String, b: Int)
implicit object FooJsonFormat extends RootJsonFormat[Foo] {
override def read(json: JsValue): Foo = {
json.asJsObject.getFields("name", "id") match {
case Seq(JsString(name), id) =>
Foo(name, id.convertTo[Int])
}
}
override def write(obj: Foo): JsValue = obj.toJson
}
This allows you to parse any arbitrary payload and pull out the fields "name" and "id" - other fields are ignored. If those fields are not guaranteed you can add something like:
case Seq(JsString(name), JsNull) =>
Foo(name, 0)
You should look at what's available in JsValue.scala - in particular JsArray may come in handy if you're getting payloads with anonymous arrays (i.e. the root is [{...}] instead of {"field":"value"...})
Spray Json doesn't support default parameters. So You cannot have a case class like
case class MyClass(a: String, b: Int = 0)
and then parse json like {"a":"foo"}
However if you make the second parameter as Option. then it works.
import spray.json._
case class MyClass(a: String, b: Option[Int] = None)
object MyProtocol extends DefaultJsonProtocol {
implicit val f = jsonFormat2(MyClass)
}
import MyProtocol.f
val mc1 = MyClass("foo", Some(10))
val strJson = mc1.toJson.toString
val strJson2 = """{"a": "foo"}"""
val mc2 = strJson2.parseJson.convertTo[MyClass]
println(mc2)
Let's say I have the next case class:
case class Person(id: String, money: BigDecimal)
object Person {
implicit val encoder: Encoder[Person] = Encoder.forProduct2("ID", "Money")(u =>
(u.id, u.money))
I want to serialize instances of the Person class to JSON, so when I evaluate the asJson from circe, I get the result in scientific notation:
{
"ID" : "123",
"VALOR_SAP" : 2.7E+7
}
Why do this happens? I think the reason is because the default to string of BigDecimal automatically format to scientific notation.
What could I do to avoid this? May be creating another type which extends from BigDecimal and overriding the toString?
I assume that you use scala.math.BigDecimal, for java.math.BigDecimal code is similar. The way to change how objects are serialized is to provide corresponding implicit Encoder object. Unfortunately both Json and JsonNumber hierachies are sealed, so there is no very clean solution but you still can use JsonNumber.fromDecimalStringUnsafe that implements toString to just return any string you passed in. So you can do something like this:
case class Person(id: String, money: BigDecimal)
object Person {
implicit final val bigDecimalAsPlainStringEncoder: Encoder[BigDecimal] = new Encoder[BigDecimal] {
final def apply(value: BigDecimal): Json = Json.fromJsonNumber(JsonNumber.fromDecimalStringUnsafe(value.bigDecimal.toPlainString))
}
implicit val encoder: Encoder[Person] = Encoder.forProduct2("ID", "Money")(u => (u.id, u.money))
}
Consider this example using Play's JSON API (play.api.libs.json):
case class FooJson(
// lots of other fields omitted
location: Option[LocationJson]
)
object FooJson {
implicit val writes = Json.writes[FooJson]
}
and
case class LocationJson(latitude: Double, longitude: Double)
object LocationJson {
implicit val writes = Json.writes[LocationJson]
}
If location is None, the resulting JSON won't have location field at all. This is fine and understadable. But if I wanted for some reason (say, to make my API more self-documenting), how can I explicitly output null in JSON?
{
"location": null
}
I also tried defining the field as location: LocationJson and passing option.orNull to it, but it does not work (scala.MatchError: null at play.api.libs.json.OWrites$$anon$2.writes). For non-custom types such as String or Double, this approach would produce null in JSON output.
So, while using Json.writes[FooJson] as shown above (or something equally simple, i.e. not having to write a custom Writes implementation), is there a clean way to include nulls in JSON?
What I'm asking is analogous to JsonInclude.Include.ALWAYS in the Jackson library (also Jackson's default behaviour). Similarly in Gson this would be trivial
(new GsonBuilder().serializeNulls().create()).
Play 2.4.4
Greg Methvin, a Play committer, wrote this answer to me in a related GitHub issue:
The JSON macros only support one way of encoding optional values,
which is to omit None values from the JSON. This is not a bug but
rather a limitation of the implementation. If you want to include
nulls you're unfortunately going to have to implement your own Writes.
I do think we should try to provide more configurability for the
macros though.
In this case, I'll let Play exclude this field when the value is null, even if it slightly sacrifices API consistency and self-documentability. It's still such a minor thing (in this particular API) that it doesn't warrant uglifying the code as much as a custom Writes would take for a case class with a dozen values.
I'm hoping they do make this more configurable in future Play versions.
Hello from the future.
As of Play 2.7, a fairly simple solution was introduced for automated JSON codecs. We can introduce the appropriate implicit value for JsonConfiguration in the scope for the Format/Reads/Writes. The following configuration will write nulls for empty Options instead of omitting the fields entirely.
import play.api.libs.json._
implicit val config = JsonConfiguration(optionHandlers = OptionHandlers.WritesNull)
implicit val residentWrites = Json.writes[Resident]
Reference
Here's a way to do it:
object MyWrites extends DefaultWrites{
override def OptionWrites[T](implicit fmt: Writes[T]): Writes[Option[T]] = new Writes[Option[T]] {
override def writes(o: Option[T]): JsValue = {
o match {
case Some(a) => Json.toJson(a)(fmt)
case None => JsNull
}
}
}
}
This will overwrite the default implementation which will not create an element. I used this in your sample code:
case class FooJson(
// ...
location: Option[LocationJson]
)
case class LocationJson(latitude: Double, longitude: Double)
object LocationJson {
implicit val writes = Json.writes[LocationJson]
}
implicit val fooJsonWriter: Writes[FooJson] = new Writes[FooJson] {
override def writes(o: FooJson): JsValue = {
JsObject(Seq(
"location" -> Json.toJson(o.location)
// Additional fields go here.
))
}
}
Json.toJson(FooJson(None))
And got this result res0: play.api.libs.json.JsValue = {"location":null}.
if we have null values then we have to add the option with members in case class which will resolve the issue
case class response(
name:String,
age: option[int]
)
object response {
implicit val format = Json.format[response]
}
Here the option is the answer for us. and if we are the JSON response for age is coming as null and this will handle the solution for us.
I am using Spray-json 1.3.1. I have the following JSON message:
{
"results": [{
... NOT IMPORTANT PART HERE ...
}],
"status": "OK"
}
Trivially, this can be deserialized to status String field via
case class Message[T](results: List[T], status: String)
with custom Protocol
object MessageProtocol extends DefaultJsonProtocol {
implicit def messageFormat[T: JsonFormat] = jsonFormat2(Message.apply[T])
}
Since status field can be one of OK, ZERO_RESULTS, OVER_QUERY_LIMIT having this field as a String makes no sense. As I am coming from
Java background I tried enums in Scala implemented as follows:
case class Message[T](results: List[T], status: Status)
object Status extends Enumeration{
type Status = Value
val OK,ZERO_RESULTS,OVER_QUERY_LIMIT, REQUEST_DENIED, INVALID_REQUEST,UNKNOWN_ERROR = Value
}
object MessageProtocol extends DefaultJsonProtocol {
implicit val statusFormat = jsonFormat(Status)
implicit def messageFormat[T: JsonFormat] = jsonFormat2(Message.apply[T])
}
What is best practice/approach to solve this?
You can simply implement your own RootJsonFormat (as an implicit in Message companion object) and override read and write functions. There you will have JsObject and you can convert it to your own case class as you want like converting the string to desired enumeration etc. You can see a sample here
I need to serialize/deserialize a Scala class with structure something like the following:
#JsonIgnoreProperties(ignoreUnknown = true, value = Array("body"))
case class Example(body: Array[Byte]) {
lazy val isNativeText = bodyIsNativeText
lazy val textEncodedBody = (if (isNativeText) new String(body, "UTF-8") else Base64.encode(body))
def this(isNativeText: Boolean, textEncodedBody: String) = this((if(isNativeText) str.getBytes("UTF-8") else Base64.decode(textEncodedBody)))
def bodyIsNativeText: Boolean = // determine if the body was natively a string or not
}
It's main member is an array of bytes, which MIGHT represent a UTF-8 encoded textual string, but might not. The primary constructor accepts an array of bytes, but there is an alternate constructor which accepts a string with a flag indicating whether this string is base64 encoded binary data, or the actual native text we want to store.
For serializing to a JSON object, I want to store the body as a native string rather than a base64-encoded string if it is native text. That's why I use #JsonIgnoreProperties to not include the body property, and instead have a textEncodedBody that gets echoed out in the JSON.
The problem comes when I try to deserialize it like so:
val e = Json.parse[Example]("""{'isNativeText': true, 'textEncodedBody': 'hello'}""")
I receive the following error:
com.codahale.jerkson.ParsingException: Invalid JSON. Needed [body],
but found [isNativeText, textEncodedBody].
Clearly, I have a constructor that will work...it just is not the default one. How can I force Jerkson to use this non-default constructor?
EDIT: I've attempted to use both the #JsonProperty and #JsonCreator annotation, but jerkson appears to disregard both of those.
EDIT2: Looking over the jerkson case class serialization source code, it looks like a case class method with the same name as its field will be used in the way that a #JsonProperty would function - that is, as a JSON getter. If I could do that, it would solve my problem. Not being super familiar with Scala, I have no idea how to do that; is it possible for a case class to have a user-defined method with the same name as one of its fields?
For reference, here is the code below that leads me to this conclusion...
private val methods = klass.getDeclaredMethods
.filter { _.getParameterTypes.isEmpty }
.map { m => m.getName -> m }.toMap
def serialize(value: A, json: JsonGenerator, provider: SerializerProvider) {
json.writeStartObject()
for (field <- nonIgnoredFields) {
val methodOpt = methods.get(field.getName)
val fieldValue: Object = methodOpt.map { _.invoke(value) }.getOrElse(field.get(value))
if (fieldValue != None) {
val fieldName = methodOpt.map { _.getName }.getOrElse(field.getName)
provider.defaultSerializeField(if (isSnakeCase) snakeCase(fieldName) else fieldName, fieldValue, json)
}
}
json.writeEndObject()
}
Correct me if I'm wrong, but it looks like Jackson/Jerkson will not support arbitrarily nested JSON. There's an example on the wiki that uses nesting, but it looks like the target class must have nested classes corresponding to the nested JSON.
Anyway, if you're not using nesting with your case classes then simply declaring a second case class and a couple implicit conversions should work just fine:
case class Example(body: Array[Byte]) {
// Note that you can just inline the body of bodyIsNativeText here
lazy val isNativeText: Boolean = // determine if the body was natively a string or not
}
case class ExampleRaw(isNativeText: Boolean, textEncodedBody: String)
implicit def exampleToExampleRaw(ex: Example) = ExampleRaw(
ex.isNativeText,
if (ex.isNativeText) new String(ex.body, "UTF-8")
else Base64.encode(ex.body)
)
implicit def exampleRawToExample(raw: ExampleRaw) = Example(
if (raw.isNativeText) raw.textEncodedBody.getBytes("UTF-8")
else Base64.decode(textEncodedBody)
)
Now you should be able to do this:
val e: Example = Json.parse[ExampleRaw](
"""{'isNativeText': true, 'textEncodedBody': 'hello'}"""
)
You could leave the original methods and annotations you added to make the JSON generation continue to work with the Example type, or you could just convert it with a cast:
generate(Example(data): ExampleRaw)
Update:
To help catch errors you might want to do something like this too:
case class Example(body: Array[Byte]) {
// Note that you can just inline the body of bodyIsNativeText here
lazy val isNativeText: Boolean = // determine if the body was natively a string or not
lazy val doNotSerialize: String = throw new Exception("Need to convert Example to ExampleRaw before serializing!")
}
That should cause an exception to be thrown if you accidentally pass an instance of Example instead of ExampleRaw to a generate call.