I am working on trying to write my own little lightweight toy Json library, and I am running into a roadblock trying to come up with an easy way to specify an Encoder/Decoder. I think Ive got a really nice dsl syntax, Im just not sure how to pull it off. I think it might be possible using Shapeless HList, but Ive never used it before, so Im drawing a blank as to how it would be done.
My thought was to chain these has calls together, and build up some sort of chain of HList[(String, J: Mapper)], and then if it is possible to have it behind the scenes try and convert a Json to a HList[J]?
Here is part of the implementation, along with how I imagine using it:
trait Mapper[J] {
def encode(j: J): Json
def decode(json: Json): Either[Json, J]
}
object Mapper {
def strict[R]: IsStrict[R] =
new IsStrict[R](true)
def lenient[R]: IsStrict[R] =
new IsStrict[R](false)
class IsStrict[R](strict: Boolean) {
def has[J: Mapper](at: String): Builder[R, J] =
???
}
class Builder[R, T](strict: Boolean, t: T) {
def has[J: Mapper](at: String): Builder[R, J] =
???
def is(decode: T => R)(encode: R => Json): Mapper[R] =
???
}
}
Mapper
.strict[Person]
.has[String]("firstName")
.has[String]("lastName")
.has[Int]("age")
.is {
case firstName :: lastName :: age :: HNil =>
new Person(firstName, lastName, age)
} { person =>
Json.Object(
"firstName" := person.firstName,
"lastName" := person.lastName,
"age" := person.age
)
}
There is a wonderful resource to learn how to use shapeless(HLIST plus LabelledGeneric) for that purpose:
Dave Gurnell´s The Type Astronaut’s Guide to Shapeless
In your case, given a product type like:
case class Person(firstName: String, lastName: String, age: Int)
The compiler should access to the names and the values of an instance of that type. The explanation about how the compiler is able to create a JSON representation at compile time is well described in the book.
In your example, you must use LabelledGeneric and try to create a generic encoder/decoder. It is a type class that creates a representation of your types as a HList where each element corresponds to a property.
For example, if you create a LabeledGeneric for your Person type
val genPerson = LabelledGeneric[Person]
the compiler infers the following type:
/*
shapeless.LabelledGeneric[test.shapeless.Person]{type Repr = shapeless.::[String with shapeless.labelled.KeyTag[Symbol with shapeless.tag.Tagged[String("firstName")],String],shapeless.::[String with shapeless.labelled.KeyTag[Symbol with shapeless.tag.Tagged[String("lastName")],String],shapeless.::[Int with shapeless.labelled.KeyTag[Symbol with shapeless.tag.Tagged[String("age")],Int],shapeless.HNil]]]}
*/
So, the names and the values are already represented using Scala types and now the compiler can derive JSON encoder/decoder instances at compile time. The code below shows the steps to create a generic JSON encoder(a summary from the chapter 5 of the book) that you can customize.
First step is to create a JSON algebraic data type:
sealed trait JsonValue
case class JsonObject(fields: List[(String, JsonValue)]) extends JsonValue
case class JsonArray(items: List[JsonValue]) extends JsonValue
case class JsonString(value: String) extends JsonValue
case class JsonNumber(value: Double) extends JsonValue
case class JsonBoolean(value: Boolean) extends JsonValue
case object JsonNull extends JsonValue
The idea behind all of this is that the compiler can take your product type and builds a JSON encoder object using the native ones.
A type class to encode your types:
trait JsonEncoder[A] {
def encode(value: A): JsonValue
}
For a first check, you can create three instances that would be necessary for the Person type:
object Instances {
implicit def StringEncoder : JsonEncoder[String] = new JsonEncoder[String] {
override def encode(value: String): JsonValue = JsonString(value)
}
implicit def IntEncoder : JsonEncoder[Double] = new JsonEncoder[Double] {
override def encode(value: Double): JsonValue = JsonNumber(value)
}
implicit def PersonEncoder(implicit strEncoder: JsonEncoder[String], numberEncoder: JsonEncoder[Double]) : JsonEncoder[Person] = new JsonEncoder[Person] {
override def encode(value: Person): JsonValue =
JsonObject("firstName" -> strEncoder.encode(value.firstName)
:: ("lastName" -> strEncoder.encode(value.firstName))
:: ("age" -> numberEncoder.encode(value.age) :: Nil))
}
}
Create an encode function that injects a JSON encoder instance:
import Instances._
def encode[A](in: A)(implicit jsonEncoder: JsonEncoder[A]) = jsonEncoder.encode(in)
val person = Person("name", "lastName", 25)
println(encode(person))
gives:
JsonObject(List((firstName,JsonString(name)), (lastName,JsonString(name)), (age,JsonNumber(25.0))))
Obviously you would need to create instances for each case class. To avoid that you need a function that returns a generic encoder:
def createObjectEncoder[A](fn: A => JsonObject): JsonObjectEncoder[A] =
new JsonObjectEncoder[A] {
def encode(value: A): JsonObject =
fn(value)
}
It needs a function A -> JsObject as parameter. The intuition behind this is that the compiler uses this function when traversing the HList representation of your type to create the type encoder, as it is described in the HList encoder function.
Then, you must create the HList encoder. That requires an implicit function to create the encoder for the HNil type and another for the HList itself.
implicit val hnilEncoder: JsonObjectEncoder[HNil] =
createObjectEncoder(hnil => JsonObject(Nil))
/* hlist encoder */
implicit def hlistObjectEncoder[K <: Symbol, H, T <: HList](
implicit witness: Witness.Aux[K],
hEncoder: Lazy[JsonEncoder[H]],
tEncoder: JsonObjectEncoder[T]): JsonObjectEncoder[FieldType[K, H] :: T] = {
val fieldName: String = witness.value.name
createObjectEncoder { hlist =>
val head = hEncoder.value.encode(hlist.head)
val tail = tEncoder.encode(hlist.tail)
JsonObject((fieldName, head) :: tail.fields)
}
}
The last thing that we have to do is to create an implicit function that injects an Encoder instance for a Person instance. It leverages the compiler implicit resolution to create a LabeledGeneric of your type and to create the encoder instance.
implicit def genericObjectEncoder[A, H](
implicit generic: LabelledGeneric.Aux[A, H],
hEncoder: Lazy[JsonObjectEncoder[H]]): JsonEncoder[A] =
createObjectEncoder { value => hEncoder.value.encode(generic.to(value))
}
You can code all these definitions inside the Instances object.
import Instances._
val person2 = Person2("name", "lastName", 25)
println(JsonEncoder[Person2].encode(person2))
prints:
JsonObject(List((firstName,JsonString(name)), (lastName,JsonString(lastName)), (age,JsonNumber(25.0))))
Note that you need to include in the HList encoder the Witness instance for Symbol. That allows to access the properties names at runtime. Remember that the LabeledGeneric of your Person type is something like:
String with KeyTag[Symbol with Tagged["firstName"], String] ::
Int with KeyTag[Symbol with Tagged["lastName"], Int] ::
Double with KeyTag[Symbol with Tagged["age"], Double] ::
The Lazy type it is necessary to create encoders for recursive types:
case class Person2(firstName: String, lastName: String, age: Double, person: Person)
val person2 = Person2("name", "lastName", 25, person)
prints:
JsonObject(List((firstName,JsonString(name)), (lastName,JsonString(lastName)), (age,JsonNumber(25.0)), (person,JsonObject(List((firstName,JsonString(name)), (lastName,JsonString(name)), (age,JsonNumber(25.0)))))))
Take a look to libraries like Circe or Spray-Json to see how they use Shapeless for codec derivation.
Try
implicit class StringOp(s: String) {
def :=[A](a: A): (String, A) = s -> a
}
implicit def strToJStr: String => Json.String = Json.String
implicit def dblToJNumber: Double => Json.Number = Json.Number
implicit def intToJNumber: Int => Json.Number = Json.Number(_)
sealed trait Json
object Json {
case class Object(fields: (scala.Predef.String, Json)*) extends Json
case class Array(items: List[Json]) extends Json
case class String(value: scala.Predef.String) extends Json
case class Number(value: Double) extends Json
case class Boolean(value: scala.Boolean) extends Json
case object Null extends Json
}
trait Mapper[J] {
def encode(j: J): Json
def decode(json: Json): Either[Json, J]
}
object Mapper {
implicit val `object`: Mapper[Json.Object] = ???
implicit val array: Mapper[Json.Array] = ???
implicit val stringJson: Mapper[Json.String] = ???
implicit val number: Mapper[Json.Number] = ???
implicit val boolean: Mapper[Json.Boolean] = ???
implicit val `null`: Mapper[Json.Null.type] = ???
implicit val json: Mapper[Json] = ???
implicit val int: Mapper[Int] = ???
implicit val string: Mapper[String] = ???
implicit val person: Mapper[Person] = ???
def strict[R]: IsStrict[R] =
new IsStrict[R](true)
def lenient[R]: IsStrict[R] =
new IsStrict[R](false)
class IsStrict[R](strict: Boolean) {
def has[A: Mapper](at: String): Builder[R, A :: HNil] =
new Builder(strict, at :: Nil)
}
class Builder[R, L <: HList](strict: Boolean, l: List[String]) {
def has[A: Mapper](at: String): Builder[R, A :: L] =
new Builder(strict, at :: l)
def is[L1 <: HList](decode: L1 => R)(encode: R => Json)(implicit
reverse: ops.hlist.Reverse.Aux[L, L1]): Mapper[R] = {
val l1 = l.reverse
???
}
}
}
Unfortunately this needs L1 to be explicitly specified for is
case class Person(firstName: String, lastName: String, age: Int)
Mapper
.strict[Person]
.has[String]("firstName")
.has[String]("lastName")
.has[Int]("age")
.is[String :: String :: Int :: HNil] {
case (firstName :: lastName :: age :: HNil) =>
new Person(firstName, lastName, age)
} { person =>
Json.Object(
"firstName" := person.firstName,
"lastName" := person.lastName,
"age" := person.age
)
}
otherwise it's Error: missing parameter type for expanded function.
The argument types of an anonymous function must be fully known.
One way to improve inference is to move implicit reverse to class Builder but this is less efficient: an HList will be reversed in every step , not only in the last step.
Another way is to introduce helper class
def is(implicit reverse: ops.hlist.Reverse[L]) = new IsHelper[reverse.Out]
class IsHelper[L1 <: HList]{
def apply(decode: L1 => R)(encode: R => Json): Mapper[R] = {
val l1 = l.reverse
???
}
}
but then apply (or other method name) should be explicit
Mapper
.strict[Person]
.has[String]("firstName")
.has[String]("lastName")
.has[Int]("age")
.is.apply {
case (firstName :: lastName :: age :: HNil) =>
new Person(firstName, lastName, age)
} { person =>
Json.Object(
"firstName" := person.firstName,
"lastName" := person.lastName,
"age" := person.age
)
}
otherwise compiler mistreats decode as reverse.
I am using the Play Framework and trying to build JSON validator for a class with abstract members. Shown below, the DataSource class is the base class which I am trying to validate the format against.
// SourceTypeConfig Trait.
trait SourceTypeConfig
final case class RDBMSConfig(...) extends SourceTypeConfig
object RDBMSConfig { implicit val fmt = Json.format[RDBMSConfig] }
final case class DirectoryConfig(
path: String,
pathType: String // Local, gcloud, azure, aws, etc.
) extends SourceTypeConfig
object DirectoryConfig { implicit val fmt = Json.format[DirectoryConfig] }
// FormatConfig trait.
trait FormatConfig
final case class SQLConfig(...) extends FormatConfig
object SQLConfig { implicit val fmt = Json.format[SQLConfig]}
final case class CSVConfig(
header: String,
inferSchema: String,
delimiter: String
) extends FormatConfig
object CSVConfig { implicit val fmt = Json.format[CSVConfig]}
// DataSource base class.
case class DataSource(
name: String,
sourceType: String,
sourceTypeConfig: SourceTypeConfig,
format: String,
formatConfig: FormatConfig
)
What I am hoping to accomplish:
val input: JsValue = Json.parse(
"""
{
"name" : "test1",
"sourceType" : "directory",
"sourceTypeConfig" : {"path" : "gs://test/path", "pathType" "google"},
"format" : "csv",
"formatConfig" : {"header" : "yes", "inferSchema" : "yes", "delimiter" : "|"}
}
"""
)
val inputResult = input.validate[DataSource]
What I am struggling with is building the DataSource object and defining its reads/writes/format. I would like it to contain a match based on the sourceType and format values that direct it to point towards the associated sourceTypeConfig and formatConfig's formats so it can parse out the JSON.
Instead of building a parser at the DataSource level, I defined parsers at the SourceConfig and FormatConfig levels, similar to what is shown below.
sealed trait SourceConfig{val sourceType: String}
object SourceConfig{
implicit val fmt = new Format[SourceConfig] {
def reads(json: JsValue): JsResult[SourceConfig] = {
def from(sourceType: String, data: JsObject): JsResult[SourceConfig] = sourceType match {
case "RDBMS" => Json.fromJson[RDBMSConfig](data)(RDBMSConfig.fmt)
case "directory" => Json.fromJson[DirectoryConfig](data)(DirectoryConfig.fmt)
case _ => JsError(s"Unknown source type: '$sourceType'")
}
for {
sourceType <- (json \ "sourceType").validate[String]
data <- json.validate[JsObject]
result <- from(sourceType, data)
} yield result
}
def writes(source: SourceConfig): JsValue =
source match {
case b: RDBMSConfig => Json.toJson(b)(RDBMSConfig.fmt)
case b: DirectoryConfig => Json.toJson(b)(DirectoryConfig.fmt)
}
}
}
Then, DataSource could be simply defined as:
object DataSource { implicit val fmt = Json.format[DataSource] }
Another option is to use play-json-derived-codecs library:
libraryDependencies += "org.julienrf" %% "play-json-derived-codecs" % "4.0.0"
import julienrf.json.derived.flat
implicit val format1: OFormat[RDBMSConfig] = Json.format[RDBMSConfig]
implicit val format2: OFormat[DirectoryConfig] = Json.format[DirectoryConfig]
implicit val format3: OFormat[SourceTypeConfig] = flat.oformat((__ \ "sourceType").format[String])
Trying to write a json format for an entity which contains a Map of Option. It throws following error
Error:(8, 68) No instance of play.api.libs.json.Format is available for scala.Predef.Map[java.lang.String, scala.Option[scala.Double]] in the implicit scope (Hint: if declared in the same file, make sure it's declared before)
Code snippet:
import play.api.libs.json.{Json, OFormat}
val a: Map[String, Option[Double]] = Map("a" -> None)
case class Person(x: Map[String, Option[Double]])
object Person {
implicit val personFormat: OFormat[Person] = Json.format[Person]
}
Json.toJson(Person(a))
You can define implicits:
import scala.collection.Map
object Person {
implicit object MapReads extends Reads[Map[String, Option[Double]]] {
def reads(jsValue: JsValue): JsResult[Map[String, Option[Double]]] = jsValue match {
case JsObject(map) => JsSuccess(
map.mapValues {
case JsNumber(d) => Some(d.toDouble)
case _ => None
}
)
case _ => JsError()
}
}
implicit object MapWrites extends Writes[Map[String, Option[Double]]] {
def writes(map: Map[String, Option[Double]]): JsValue =
JsObject(map.mapValues(optd => JsNumber(optd.getOrElse[Double](0.0))))
}
implicit val personFormat: OFormat[Person] = Json.format[Person]
}
println(Json.toJson(Person(a)))//{"x":{"a":0}}
Based on answer.
The problem seems to be the inability of play-json macros to work with nested type variables:
Map[String, Option[Double]]
You could use an intermediate type wrapping Option
import play.api.libs.json.{Json, OFormat}
case class OptionDouble(value: Option[Double])
case class Person(x: Map[String, OptionDouble])
implicit val optionDoubleFormat = Json.format[OptionDouble]
implicit val personFormat = Json.format[Person]
val a: Map[String, OptionDouble] = Map("a" -> OptionDouble(None))
Json.toJson(Person(a))
Another option could be to write the formatter by hand.
I am trying to use akka-http-spray-json 10.0.9
My model:
case class Person(id: Long, name: String, age: Int)
I get json string jsonStr with list of persons and try to parse it:
implicit val personFormat: RootJsonFormat[Person] = jsonFormat3(Person)
val json = jsonStr.parseJson
val persons = json.convertTo[Seq[Person]]
Error:
Object expected in field 'id'
Probably i need to create implicit object extends RootJsonFormat[List[Person]] and override read and write methods.
implicit object personsListFormat extends RootJsonFormat[List[Person]] {
override def write(persons: List[Person]) = ???
override def read(json: JsValue) = {
// Maybe something like
// json.map(_.convertTo[Person])
// But there is no map or similar method :(
}
}
P.S. Sorry for my english, it's not my native.
UPD
jsonStr:
[ {"id":6,"name":"Martin Ordersky","age":50}, {"id":8,"name":"Linus Torwalds","age":43}, {"id":9,"name":"James Gosling","age":45}, {"id":10,"name":"Bjarne Stroustrup","age":59} ]
I get perfectly expected results with:
import spray.json._
object MyJsonProtocol extends DefaultJsonProtocol {
implicit val personFormat: JsonFormat[Person] = jsonFormat3(Person)
}
import MyJsonProtocol._
val jsonStr = """[{"id":1,"name":"john","age":40}]"""
val json = jsonStr.parseJson
val persons = json.convertTo[List[Person]]
persons.foreach(println)
I'm using my own implicit implementation of JSON serializer and deserializer for my case object
My case class looks like (it's just a code snippet)
sealed trait MyTrait
case object MyCaseClass extends MyTrait
I want to write my own ser. and deser. of JSON for MyTrait
implicit val myTraitFormat = new JsonFormat[MyTrait] {
override def read(json: JsValue): MyTrait = json.asJsObject.getFields("value") match {
case Seq(JsString("TEST")) ⇒ MyCaseClass
case _ ⇒ throw new DeserializationException(s"$json is not a valid extension of my trait")
}
override def write(myTrait: MyTrait): JsValue = {
myTrait match {
case MyCaseClass => JsObject("value" -> JsString("TEST"))
}
}
}
Now my test is failing by throwing DeserializationException:
"The my JSON format" when {
"deserializing a JSON" must {
"return correct object" in {
val json = """{"value": "TEST"}""".asJson
json.convertTo[MyTrait] must equal (MyCaseClass)
}
}
}
Obviously json.asJsObject.getFields("value")can not be matched to Seq(JsString("TEST")). Maybe this is related to using traits?
But I have found example on official spray-json site https://github.com/spray/spray-json#providing-jsonformats-for-other-types
Any ideas how to properly match on field in JsObject?
Thanks!
Best
Write variable name instead of string:
override def read(json: JsValue): MyCaseClass = json.asJsObject.getFields("value") match {
case Seq(JsString(value)) ⇒ MyCaseClass
case _ ⇒ throw new DeserializationException(s"$json is not a valid case class")
}
Does that work for you?
Update:
Yes, your test is wrong. I tried it with the following (note parseJson instead of asJson) and it worked:
scala> val json = """{"value": "TEST"}""".parseJson
json: spray.json.JsValue = {"value":"TEST"}
scala> json.convertTo[MyCaseClass]
res2: MyCaseClass = MyCaseClass()
Update 2: Tried it with trait:
scala> import spray.json._
import spray.json._
scala> import spray.json.DefaultJsonProtocol._
import spray.json.DefaultJsonProtocol._
[Your code]
scala> val json = """{"value": "TEST"}""".parseJson
json: spray.json.JsValue = {"value":"TEST"}
scala> json.convertTo[MyTrait]
res1: MyTrait = MyCaseClass