Using Shapeless HList to easily build Json Decoder - json

Question

I am working on trying to write my own little lightweight toy Json library, and I am running into a roadblock trying to come up with an easy way to specify an Encoder/Decoder. I think Ive got a really nice dsl syntax, Im just not sure how to pull it off. I think it might be possible using Shapeless HList, but Ive never used it before, so Im drawing a blank as to how it would be done.
My thought was to chain these has calls together, and build up some sort of chain of HList[(String, J: Mapper)], and then if it is possible to have it behind the scenes try and convert a Json to a HList[J]?
Here is part of the implementation, along with how I imagine using it:
trait Mapper[J] {
def encode(j: J): Json
def decode(json: Json): Either[Json, J]
}
object Mapper {
def strict[R]: IsStrict[R] =
new IsStrict[R](true)
def lenient[R]: IsStrict[R] =
new IsStrict[R](false)
class IsStrict[R](strict: Boolean) {
def has[J: Mapper](at: String): Builder[R, J] =
???
}
class Builder[R, T](strict: Boolean, t: T) {
def has[J: Mapper](at: String): Builder[R, J] =
???
def is(decode: T => R)(encode: R => Json): Mapper[R] =
???
}
}
Mapper
.strict[Person]
.has[String]("firstName")
.has[String]("lastName")
.has[Int]("age")
.is {
case firstName :: lastName :: age :: HNil =>
new Person(firstName, lastName, age)
} { person =>
Json.Object(
"firstName" := person.firstName,
"lastName" := person.lastName,
"age" := person.age
)
}

Answer 1

There is a wonderful resource to learn how to use shapeless(HLIST plus LabelledGeneric) for that purpose:
Dave Gurnell´s The Type Astronaut’s Guide to Shapeless
In your case, given a product type like:
case class Person(firstName: String, lastName: String, age: Int)
The compiler should access to the names and the values of an instance of that type. The explanation about how the compiler is able to create a JSON representation at compile time is well described in the book.
In your example, you must use LabelledGeneric and try to create a generic encoder/decoder. It is a type class that creates a representation of your types as a HList where each element corresponds to a property.
For example, if you create a LabeledGeneric for your Person type
val genPerson = LabelledGeneric[Person]
the compiler infers the following type:
/*
shapeless.LabelledGeneric[test.shapeless.Person]{type Repr = shapeless.::[String with shapeless.labelled.KeyTag[Symbol with shapeless.tag.Tagged[String("firstName")],String],shapeless.::[String with shapeless.labelled.KeyTag[Symbol with shapeless.tag.Tagged[String("lastName")],String],shapeless.::[Int with shapeless.labelled.KeyTag[Symbol with shapeless.tag.Tagged[String("age")],Int],shapeless.HNil]]]}
*/
So, the names and the values are already represented using Scala types and now the compiler can derive JSON encoder/decoder instances at compile time. The code below shows the steps to create a generic JSON encoder(a summary from the chapter 5 of the book) that you can customize.
First step is to create a JSON algebraic data type:
sealed trait JsonValue
case class JsonObject(fields: List[(String, JsonValue)]) extends JsonValue
case class JsonArray(items: List[JsonValue]) extends JsonValue
case class JsonString(value: String) extends JsonValue
case class JsonNumber(value: Double) extends JsonValue
case class JsonBoolean(value: Boolean) extends JsonValue
case object JsonNull extends JsonValue
The idea behind all of this is that the compiler can take your product type and builds a JSON encoder object using the native ones.
A type class to encode your types:
trait JsonEncoder[A] {
def encode(value: A): JsonValue
}
For a first check, you can create three instances that would be necessary for the Person type:
object Instances {
implicit def StringEncoder : JsonEncoder[String] = new JsonEncoder[String] {
override def encode(value: String): JsonValue = JsonString(value)
}
implicit def IntEncoder : JsonEncoder[Double] = new JsonEncoder[Double] {
override def encode(value: Double): JsonValue = JsonNumber(value)
}
implicit def PersonEncoder(implicit strEncoder: JsonEncoder[String], numberEncoder: JsonEncoder[Double]) : JsonEncoder[Person] = new JsonEncoder[Person] {
override def encode(value: Person): JsonValue =
JsonObject("firstName" -> strEncoder.encode(value.firstName)
:: ("lastName" -> strEncoder.encode(value.firstName))
:: ("age" -> numberEncoder.encode(value.age) :: Nil))
}
}
Create an encode function that injects a JSON encoder instance:
import Instances._
def encode[A](in: A)(implicit jsonEncoder: JsonEncoder[A]) = jsonEncoder.encode(in)
val person = Person("name", "lastName", 25)
println(encode(person))
gives:
JsonObject(List((firstName,JsonString(name)), (lastName,JsonString(name)), (age,JsonNumber(25.0))))
Obviously you would need to create instances for each case class. To avoid that you need a function that returns a generic encoder:
def createObjectEncoder[A](fn: A => JsonObject): JsonObjectEncoder[A] =
new JsonObjectEncoder[A] {
def encode(value: A): JsonObject =
fn(value)
}
It needs a function A -> JsObject as parameter. The intuition behind this is that the compiler uses this function when traversing the HList representation of your type to create the type encoder, as it is described in the HList encoder function.
Then, you must create the HList encoder. That requires an implicit function to create the encoder for the HNil type and another for the HList itself.
implicit val hnilEncoder: JsonObjectEncoder[HNil] =
createObjectEncoder(hnil => JsonObject(Nil))
/* hlist encoder */
implicit def hlistObjectEncoder[K <: Symbol, H, T <: HList](
implicit witness: Witness.Aux[K],
hEncoder: Lazy[JsonEncoder[H]],
tEncoder: JsonObjectEncoder[T]): JsonObjectEncoder[FieldType[K, H] :: T] = {
val fieldName: String = witness.value.name
createObjectEncoder { hlist =>
val head = hEncoder.value.encode(hlist.head)
val tail = tEncoder.encode(hlist.tail)
JsonObject((fieldName, head) :: tail.fields)
}
}
The last thing that we have to do is to create an implicit function that injects an Encoder instance for a Person instance. It leverages the compiler implicit resolution to create a LabeledGeneric of your type and to create the encoder instance.
implicit def genericObjectEncoder[A, H](
implicit generic: LabelledGeneric.Aux[A, H],
hEncoder: Lazy[JsonObjectEncoder[H]]): JsonEncoder[A] =
createObjectEncoder { value => hEncoder.value.encode(generic.to(value))
}
You can code all these definitions inside the Instances object.
import Instances._
val person2 = Person2("name", "lastName", 25)
println(JsonEncoder[Person2].encode(person2))
prints:
JsonObject(List((firstName,JsonString(name)), (lastName,JsonString(lastName)), (age,JsonNumber(25.0))))
Note that you need to include in the HList encoder the Witness instance for Symbol. That allows to access the properties names at runtime. Remember that the LabeledGeneric of your Person type is something like:
String with KeyTag[Symbol with Tagged["firstName"], String] ::
Int with KeyTag[Symbol with Tagged["lastName"], Int] ::
Double with KeyTag[Symbol with Tagged["age"], Double] ::
The Lazy type it is necessary to create encoders for recursive types:
case class Person2(firstName: String, lastName: String, age: Double, person: Person)
val person2 = Person2("name", "lastName", 25, person)
prints:
JsonObject(List((firstName,JsonString(name)), (lastName,JsonString(lastName)), (age,JsonNumber(25.0)), (person,JsonObject(List((firstName,JsonString(name)), (lastName,JsonString(name)), (age,JsonNumber(25.0)))))))
Take a look to libraries like Circe or Spray-Json to see how they use Shapeless for codec derivation.

Answer 2

Try
implicit class StringOp(s: String) {
def :=[A](a: A): (String, A) = s -> a
}
implicit def strToJStr: String => Json.String = Json.String
implicit def dblToJNumber: Double => Json.Number = Json.Number
implicit def intToJNumber: Int => Json.Number = Json.Number(_)
sealed trait Json
object Json {
case class Object(fields: (scala.Predef.String, Json)*) extends Json
case class Array(items: List[Json]) extends Json
case class String(value: scala.Predef.String) extends Json
case class Number(value: Double) extends Json
case class Boolean(value: scala.Boolean) extends Json
case object Null extends Json
}
trait Mapper[J] {
def encode(j: J): Json
def decode(json: Json): Either[Json, J]
}
object Mapper {
implicit val `object`: Mapper[Json.Object] = ???
implicit val array: Mapper[Json.Array] = ???
implicit val stringJson: Mapper[Json.String] = ???
implicit val number: Mapper[Json.Number] = ???
implicit val boolean: Mapper[Json.Boolean] = ???
implicit val `null`: Mapper[Json.Null.type] = ???
implicit val json: Mapper[Json] = ???
implicit val int: Mapper[Int] = ???
implicit val string: Mapper[String] = ???
implicit val person: Mapper[Person] = ???
def strict[R]: IsStrict[R] =
new IsStrict[R](true)
def lenient[R]: IsStrict[R] =
new IsStrict[R](false)
class IsStrict[R](strict: Boolean) {
def has[A: Mapper](at: String): Builder[R, A :: HNil] =
new Builder(strict, at :: Nil)
}
class Builder[R, L <: HList](strict: Boolean, l: List[String]) {
def has[A: Mapper](at: String): Builder[R, A :: L] =
new Builder(strict, at :: l)
def is[L1 <: HList](decode: L1 => R)(encode: R => Json)(implicit
reverse: ops.hlist.Reverse.Aux[L, L1]): Mapper[R] = {
val l1 = l.reverse
???
}
}
}
Unfortunately this needs L1 to be explicitly specified for is
case class Person(firstName: String, lastName: String, age: Int)
Mapper
.strict[Person]
.has[String]("firstName")
.has[String]("lastName")
.has[Int]("age")
.is[String :: String :: Int :: HNil] {
case (firstName :: lastName :: age :: HNil) =>
new Person(firstName, lastName, age)
} { person =>
Json.Object(
"firstName" := person.firstName,
"lastName" := person.lastName,
"age" := person.age
)
}
otherwise it's Error: missing parameter type for expanded function.
The argument types of an anonymous function must be fully known.
One way to improve inference is to move implicit reverse to class Builder but this is less efficient: an HList will be reversed in every step , not only in the last step.
Another way is to introduce helper class
def is(implicit reverse: ops.hlist.Reverse[L]) = new IsHelper[reverse.Out]
class IsHelper[L1 <: HList]{
def apply(decode: L1 => R)(encode: R => Json): Mapper[R] = {
val l1 = l.reverse
???
}
}
but then apply (or other method name) should be explicit
Mapper
.strict[Person]
.has[String]("firstName")
.has[String]("lastName")
.has[Int]("age")
.is.apply {
case (firstName :: lastName :: age :: HNil) =>
new Person(firstName, lastName, age)
} { person =>
Json.Object(
"firstName" := person.firstName,
"lastName" := person.lastName,
"age" := person.age
)
}
otherwise compiler mistreats decode as reverse.