I am using Circe for json operations. I have added custom encoders and decoders to handle some of the types, like Joda Time.
While parsing DateTime, I want to allow multiple formats to be passed.
For eg. dd-MM-yyyy'T'HH:mm:ss'Z' and dd-MM-yyyy'T'HH:mm:ss.SSS'Z'
I have defined my decoder like below:
val dateTimeFormat = DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss'Z'")
val dateTimeFormatWithMillis = DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")
implicit val jodaDateTimeFormat: Encoder[DateTime] with Decoder[DateTime] = new Encoder[DateTime] with Decoder[DateTime] {
override def apply(a: DateTime): Json = Encoder.encodeString(a.toString("yyyy-MM-dd'T'HH:mm:ss'Z'"))
override def apply(c: HCursor): Result[DateTime] = Decoder.decodeString.map { x =>
DateTime.parse(x, dateTimeFormat)
}.apply(c)
}
Now if i input a datetime string matching the dateTimeFormat, then the decoding will work, but if I pass the datetime in dateTimeFormatWithMillis, it will fail to process.
I know that I can use the DateTimeFormatterBuilder to add multiple parsers and process it, however, I was wondering if there is a way in Circe to chain multiple decoders to try one after another until it succeeds or reached end of chain?
You can use Decoder#or to combine decoders so that the second one is tried in case the first one fails.
Here is a working example:
import org.joda.time.DateTime
import org.joda.time.format.{DateTimeFormat, DateTimeFormatter}
import io.circe.{Decoder, Encoder}
import io.circe.parser.decode
import scala.util.Try
val dateTimeFormat = DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss'Z'")
val dateTimeFormatWithMillis = DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")
/** Creates a decoder that decodes a [[DateTime]] using the provided format. */
def dateTimeFormatDecoder(format: DateTimeFormatter): Decoder[DateTime] =
Decoder[String].emapTry(str => Try(DateTime.parse(str, format)))
/** [[Decoder]] for the first format (without milliseconds). */
val dateTimeWithoutMillisDecoder: Decoder[DateTime] =
dateTimeFormatDecoder(dateTimeFormat)
/** [[Decoder]] for the second format (with milliseconds). */
val dateTimeWithMillisDecoder: Decoder[DateTime] =
dateTimeFormatDecoder(dateTimeFormatWithMillis)
/** Encodes a [[DateTime]] using `Encoder[String].contramap(...)`, which is
* perhaps a slightly more idiomatic version of
* `Encoder.encodeString(a.toString("yyyy-MM-dd'T'HH:mm:ss'Z'"))` */
implicit val jodaDateTimeEncoder: Encoder[DateTime] =
Encoder[String].contramap(_.toString("yyyy-MM-dd'T'HH:mm:ss'Z'"))
implicit val jodaDateTimeDecoder: Decoder[DateTime] =
dateTimeWithoutMillisDecoder or dateTimeWithMillisDecoder
println(decode[DateTime](""" "2001-02-03T04:05:06Z" """))
println(decode[DateTime](""" "2001-02-03T04:05:06.789Z" """))
Note that the Encoder and Decoder have been separated since Decoder#or returns a Decoder, which wouldn’t work with a combined class (i.e. Encoder[DateTime] with Decoder[DateTime]).
Also, the DateTime.parse calls have been wrapped with Decoder#emapTry because the or combinator (and in general all Decoder combinators) expects to be dealing with Either values, not exceptions.
Related
I have a problem with declaring of generic method in Play 2.6 application that converts JSON to instance of one of the case class models.
All models declared with helper objects and formatters:
import play.api.libs.json.{Json, OFormat}
case class Shot(id: Long, likes_count: Long)
object Shot {
implicit val format: OFormat[Shot] = Json.format[Shot]
}
val s1: Shot = Json.toJson(f).as[Shot] // Works great
def testJsonGeneric[T](js: JsValue)(implicit ev: OFormat[T]): T = {
js.as[T](ev)
}
val s2: Shot = testJsonGeneric(Json.toJson(f)) // could not find implicit value for parameter ev: play.api.libs.json.OFormat[T]. Compilation failed
The last line of code throws
could not find implicit value for parameter ev: play.api.libs.json.OFormat[T]
But if I call my generic method like this (with explicit formatter) it works just fine:
val s2: Shot = testJsonGeneric(Json.toJson(f))(Shot.format)
However, it looks like if I expect my JSON to return a list of objects I have to define an extra formatter for List[Shot] to pass explicitly to the method when default Play's json.as[List[Shot]] could easily allow me to do this with a single existing formatter like the one that already defined in the helper object.
So, is it even possible to provide formatters implicitly for generic type T in my case?
Thank you
You can definitely do this, you just have to change the declaration a bit.
Move the case class and companion declaration outside the method, and then explicitly import Shot._ to bring the implicit in scope:
import play.api.libs.json.{JsValue, Json, OFormat}
object Foo {
case class Shot(id: Long, likes_count: Long)
object Shot {
implicit def format: OFormat[Shot] = Json.format[Shot]
}
def main(args: Array[String]): Unit = {
import Shot._
val f = Shot(1, 2)
def testJsonGeneric[T](js: JsValue)(implicit ev: OFormat[T]): T = {
js.as[T](ev)
}
val s2: Shot = testJsonGeneric(Json.toJson(f))
}
}
I would like to design a base trait/class in Scala that can produce the following json:
trait GenericResource {
val singularName: String
val pluralName: String
}
I would inherit this trait in a case class:
case class Product(name: String) extends GenericResource {
override val singularName = "product"
override val pluralName = "products"
}
val car = Product("car")
val jsonString = serialize(car)
the output should look like: {"product":{"name":"car"}}
A Seq[Product] should produce {"products":[{"name":"car"},{"name":"truck"}]} etc...
I'm struggling with the proper abstractions to accomplish this. I am open to solutions using any JSON library (available in Scala).
Here's about the simplest way I can think of to do the singular part generically with circe:
import io.circe.{ Decoder, Encoder, Json }
import io.circe.generic.encoding.DerivedObjectEncoder
trait GenericResource {
val singularName: String
val pluralName: String
}
object GenericResource {
implicit def encodeResource[A <: GenericResource](implicit
derived: DerivedObjectEncoder[A]
): Encoder[A] = Encoder.instance { a =>
Json.obj(a.singularName -> derived(a))
}
}
And then if you have some case class extending GenericResource like this:
case class Product(name: String) extends GenericResource {
val singularName = "product"
val pluralName = "products"
}
You can do this (assuming all the members of the case class are encodeable):
scala> import io.circe.syntax._
import io.circe.syntax._
scala> Product("car").asJson.noSpaces
res0: String = {"product":{"name":"car"}}
No boilerplate, no extra imports, etc.
The Seq case is a little trickier, since circe automatically provides a Seq[A] encoder for any A that has an Encoder, but it doesn't do what you want—it just encodes the items and sticks them in a JSON array. You can write something like this:
implicit def encodeResources[A <: GenericResource](implicit
derived: DerivedObjectEncoder[A]
): Encoder[Seq[A]] = Encoder.instance {
case values # (head +: _) =>
Json.obj(head.pluralName -> Encoder.encodeList(derived)(values.toList))
case Nil => Json.obj()
}
And use it like this:
scala> Seq(Product("car"), Product("truck")).asJson.noSpaces
res1: String = {"products":[{"name":"car"},{"name":"truck"}]}
But you can't just stick it in the companion object and expect everything to work—you have to put it somewhere and import it when you need it (otherwise it has the same priority as the default Seq[A] instances).
Another issue with this encodeResources implementation is that it just returns an empty object if the Seq is empty:
scala> Seq.empty[Product].asJson.noSpaces
res2: String = {}
This is because the plural name is attached to the resource at the instance level, and if you don't have an instance there's no way to get it (short of reflection). You could of course conjure up a fake instance by passing nulls to the constructor or whatever, but that seems out of the scope of this question.
This issue (the resource names being attached to instances) is also going to be trouble if you need to decode this JSON you've encoded. If that is the case, I'd suggest considering a slightly different approach where you have something like a GenericResourceCompanion trait that you mix into the companion object for the specific resource type, and to indicate the names there. If that's not an option, you're probably stuck with reflection or fake instances, or both (but again, probably not in scope for this question).
I am building HTTP service with scala-play. I want to serve the JSON response.
To serve JSON response, I created case classes and created instance of Writes[T] class which is similar to following
case class Point(X: Double, Y: Double)
implicit val pointWrites = new Writes[Point] {
def writes(point: Point) = Json.obj(
"X" -> point.X,
"Y" -> point.Y
)
}
I can server response like: val json = Json.toJson(Point(10, 20))
I followed above approach from play documentation.
Why do I need to define class and create Writes[T] instance for serve JSON response.
If I have to create JSON for dozens of classes, do I have to define all classes and create instances for Writes[T]?
I can easily convert any class data to JSON with following method:
def getJSON(data: Map[String, String]): JsObject = {
JsObject(data.map { el => el._1 -> JsString(el._2) })
}
All I need do is create Map[String, String] for any class to serve JSON.
Why play documentation recommends to define case class and create instance of Writes[T] to create JSON representation?
Is there any other way to serve create JSON in scala? I want to keep response time minimal so that JSON creation must be lightening fast.
You don't need to create the Writes instance. Play already have a default formatter for all the regular types (Integer/String/Boolean..)
You just need to write the case class and add an implicit formatter in the companion object.
See json.scala format[A] description from play:
/**
* Creates a Format[T] by resolving case class fields & required implicits at COMPILE-time
*
* If any missing implicit is discovered, compiler will break with corresponding error.
* {{{
* import play.api.libs.json.Json
*
* case class User(name: String, age: Int)
*
* implicit val userWrites = Json.format[User]
* // macro-compiler replaces Json.format[User] by injecting into compile chain
* // the exact code you would write yourself. This is strictly equivalent to:
* implicit val userWrites = (
* (__ \ 'name).format[String] and
* (__ \ 'age).format[Int]
* )(User.apply, unlift(User.unapply))
* }}}
*/
def format[A] = macro JsMacroImpl.formatImpl[A]
Back to your example, you only need:
case class Point(X: Double, Y: Double)
object Point{
implicit val fmtJson = Json.format[Point]
}
to get the json text just do:
val json = Point.fmtJson.writes(Point(10, 20)).toString()
will result: json = {"X":10,"Y":20}
For the validate method on request.body it matches the attribute name and value type of the json object to those defined in the model definition. Now if I were to add an extra attribute to the json object and try to validate it, it passes as a JsSuccess when it shouldn't.
{
"Name": "Bob",
"Age": 20,
"Random_Field_Not_Defined_in_Models": "Test"
}
My Person Class is defined as follows
case class Person(name: String, age: Int)
I'm assuming you've been using the built-in Reads[T] or Format[T] converters that Play gives you via Json.reads[T], e.g.:
import play.api.libs.json._
val standardReads = Json.reads[Person]
While these are super-handy, if you need additional validation, you'll have to define a custom Reads[Person] class; but fortunately we can still leverage the built-in JSON-to-case-class macro to do the basic checking and conversion, and then add an extra layer of custom checks if things seem OK:
val standardReads = Json.reads[Person]
val strictReads = new Reads[Person] {
val expectedKeys = Set("name", "age")
def reads(jsv:JsValue):JsResult[Person] = {
standardReads.reads(jsv).flatMap { person =>
checkUnwantedKeys(jsv, person)
}
}
private def checkUnwantedKeys(jsv:JsValue, p:Person):JsResult[Person] = {
val obj = jsv.asInstanceOf[JsObject]
val keys = obj.keys
val unwanted = keys.diff(expectedKeys)
if (unwanted.isEmpty) {
JsSuccess(p)
} else {
JsError(s"Keys: ${unwanted.mkString(",")} found in the incoming JSON")
}
}
}
Note how we utilize standardReads first, to make sure we're dealing with something that can be converted to a Person. No need to reinvent the wheel here.
We use flatMap to effectively short-circuit the conversion if we get a JsError from standardReads - i.e. we only call checkUnwantedKeys if needed.
checkUnwantedKeys just uses the fact that a JsObject is really just a wrapper around a Map, so we can easily check the names of the keys against a whitelist.
Note that you could also write that flatMap using a for-comprehension, which starts to look a lot cleaner if you need even more checking stages:
for {
p <- standardReads.reads(jsv)
r1 <- checkUnexpectedFields(jsv, p)
r2 <- checkSomeOtherStuff(jsv, r1)
r3 <- checkEvenMoreStuff(jsv, r2)
} yield r3
If you want to avoid too much boilerplate it is possible to make a more generic solution using a little bit of scala reflection:
import play.api.libs.json._
import scala.reflect.runtime.universe._
def checkedReads[T](underlyingReads: Reads[T])(implicit typeTag: TypeTag[T]): Reads[T] = new Reads[T] {
def classFields[T: TypeTag]: Set[String] = typeOf[T].members.collect {
case m: MethodSymbol if m.isCaseAccessor => m.name.decodedName.toString
}.toSet
def reads(json: JsValue): JsResult[T] = {
val caseClassFields = classFields[T]
json match {
case JsObject(fields) if (fields.keySet -- caseClassFields).nonEmpty =>
JsError(s"Unexpected fields provided: ${(fields.keySet -- caseClassFields).mkString(", ")}")
case _ => underlyingReads.reads(json)
}
}
}
Then you can specify your reads instances as:
implicit val reads = checkedReads(Json.reads[Person])
This leverages a fair bit of Scala type magic and also the reflection library (that lets you look at fields on classes).
Rather than relying on a fixed set of fields the classFields method gets all of the fields dynamically for the case class (type param T). It looks at all of the members and collects only the case class accessors (otherwise we'd pick up methods like toString). It returns a Set[String] of field names.
You'll notice that the checkedReads takes an implicit TypeTag[T]. This is supplied by the compiler at compile time and used by the typeOf method.
The remaining code is fairly self explanatory. If the incoming json matches our first case (it is a JsObject and there are fields not on the case class) then we return a JsError. Otherwise we pass it on to the underlying reader.
I'm looking for suggestions or libraries that can help me convert JSON (with nested structure) from one format to another in Scala.
I saw there are a few JavaScript and Java based solutions. Anything in Scala ?
I really like the Play JSON library. It's API is very clean and it's very fast even if some parts have a slightly steeper learning curve. You can also use the Play JSON library even if you aren't using the rest of Play.
https://playframework.com/documentation/2.3.x/ScalaJson
To convert JSON to scala objects (and vice versa), Play uses implicits. There is a Reads type which specifies how to convert JSON to a scala type, and a Writes type which specifies how to convert a scala object to JSON.
For example:
case class Foo(a: Int, b: String)
There are a few different routes you can take to convert Foo to JSON. If your object is simple (like Foo), Play JSON can create a conversion function for you:
implicit val fooReads = Json.reads[Foo]
or you can create a custom conversion function if you want more control or if your type is more complex. The below examples uses the name id for the property a in Foo:
implicit val fooReads = (
(__ \ "id").read[Int] ~
(__ \ "name").read[String]
)(Foo)
The Writes type has similar capabilities:
implicit val fooWrites = Json.writes[Foo]
or
implicit val fooWrites = (
(JsPath \ "id").write[Int] and
(JsPath \ "name").write[String]
)(unlift(Foo.unapply))
You can read more about Reads/Writes (and all the imports you will need) here: https://playframework.com/documentation/2.3.x/ScalaJsonCombinators
You can also transform your JSON without mapping JSON to/from scala types. This is fast and often requires less boilerplate. A simple example:
import play.api.libs.json._
// Only take a single branch from the input json
// This transformer takes the entire JSON subtree pointed to by
// key bar (no matter what it is)
val pickFoo = (__ \ 'foo).json.pickBranch
// Parse JSON from a string and apply the transformer
val input = """{"foo": {"id": 10, "name": "x"}, "foobar": 100}"""
val baz: JsValue = Json.parse(input)
val foo: JsValue = baz.transform(pickFoo)
You can read more about transforming JSON directly here: https://playframework.com/documentation/2.3.x/ScalaJsonTransformers
You can use Json4s Jackson. With PlayJson, you have to write Implicit conversions for all the case classes. If the no. of classes are small, and will not have frequent changes while development, PlayJson seems to be okay. But, if the case classes are more, I recommend using json4s.
You need to add implicit conversion for different types, so that json4s will understand while converting to json.
You can add the below dependency to your project to get json4s-jackson
"org.json4s" %% "json4s-jackson" % "3.2.11"
A sample code is given below (with both serialization and deserialization):
import java.util.Date
import java.text.SimpleDateFormat
import org.json4s.DefaultFormats
import org.json4s.jackson.JsonMethods._
import org.json4s.jackson.{Serialization}
/**
* Created by krishna on 19/5/15.
*/
case class Parent(id:Long, name:String, children:List[Child])
case class Child(id:Long, name:String, simple: Simple)
case class Simple(id:Long, name:String, date:Date)
object MainClass extends App {
implicit val formats = (new DefaultFormats {
override def dateFormatter = new SimpleDateFormat("yyyy-MM-dd")
}.preservingEmptyValues)
val d = new Date()
val simple = Simple(1L, "Simple", d)
val child1 = Child(1L, "Child1", simple)
val child2 = Child(2L, "Child2", simple)
val parent = Parent(1L, "Parent", List(child1, child2))
//Conversion from Case Class to Json
val json = Serialization.write(parent)
println(json)
//Conversion from Json to Case Class
val parentFromJson = parse(json).extract[Parent]
println(parentFromJson)
}