I am working on trying to write my own little lightweight toy Json library, and I am running into a roadblock trying to come up with an easy way to specify an Encoder/Decoder. I think Ive got a really nice dsl syntax, Im just not sure how to pull it off. I think it might be possible using Shapeless HList, but Ive never used it before, so Im drawing a blank as to how it would be done.
My thought was to chain these has calls together, and build up some sort of chain of HList[(String, J: Mapper)], and then if it is possible to have it behind the scenes try and convert a Json to a HList[J]?
Here is part of the implementation, along with how I imagine using it:
trait Mapper[J] {
def encode(j: J): Json
def decode(json: Json): Either[Json, J]
}
object Mapper {
def strict[R]: IsStrict[R] =
new IsStrict[R](true)
def lenient[R]: IsStrict[R] =
new IsStrict[R](false)
class IsStrict[R](strict: Boolean) {
def has[J: Mapper](at: String): Builder[R, J] =
???
}
class Builder[R, T](strict: Boolean, t: T) {
def has[J: Mapper](at: String): Builder[R, J] =
???
def is(decode: T => R)(encode: R => Json): Mapper[R] =
???
}
}
Mapper
.strict[Person]
.has[String]("firstName")
.has[String]("lastName")
.has[Int]("age")
.is {
case firstName :: lastName :: age :: HNil =>
new Person(firstName, lastName, age)
} { person =>
Json.Object(
"firstName" := person.firstName,
"lastName" := person.lastName,
"age" := person.age
)
}
There is a wonderful resource to learn how to use shapeless(HLIST plus LabelledGeneric) for that purpose:
Dave Gurnell´s The Type Astronaut’s Guide to Shapeless
In your case, given a product type like:
case class Person(firstName: String, lastName: String, age: Int)
The compiler should access to the names and the values of an instance of that type. The explanation about how the compiler is able to create a JSON representation at compile time is well described in the book.
In your example, you must use LabelledGeneric and try to create a generic encoder/decoder. It is a type class that creates a representation of your types as a HList where each element corresponds to a property.
For example, if you create a LabeledGeneric for your Person type
val genPerson = LabelledGeneric[Person]
the compiler infers the following type:
/*
shapeless.LabelledGeneric[test.shapeless.Person]{type Repr = shapeless.::[String with shapeless.labelled.KeyTag[Symbol with shapeless.tag.Tagged[String("firstName")],String],shapeless.::[String with shapeless.labelled.KeyTag[Symbol with shapeless.tag.Tagged[String("lastName")],String],shapeless.::[Int with shapeless.labelled.KeyTag[Symbol with shapeless.tag.Tagged[String("age")],Int],shapeless.HNil]]]}
*/
So, the names and the values are already represented using Scala types and now the compiler can derive JSON encoder/decoder instances at compile time. The code below shows the steps to create a generic JSON encoder(a summary from the chapter 5 of the book) that you can customize.
First step is to create a JSON algebraic data type:
sealed trait JsonValue
case class JsonObject(fields: List[(String, JsonValue)]) extends JsonValue
case class JsonArray(items: List[JsonValue]) extends JsonValue
case class JsonString(value: String) extends JsonValue
case class JsonNumber(value: Double) extends JsonValue
case class JsonBoolean(value: Boolean) extends JsonValue
case object JsonNull extends JsonValue
The idea behind all of this is that the compiler can take your product type and builds a JSON encoder object using the native ones.
A type class to encode your types:
trait JsonEncoder[A] {
def encode(value: A): JsonValue
}
For a first check, you can create three instances that would be necessary for the Person type:
object Instances {
implicit def StringEncoder : JsonEncoder[String] = new JsonEncoder[String] {
override def encode(value: String): JsonValue = JsonString(value)
}
implicit def IntEncoder : JsonEncoder[Double] = new JsonEncoder[Double] {
override def encode(value: Double): JsonValue = JsonNumber(value)
}
implicit def PersonEncoder(implicit strEncoder: JsonEncoder[String], numberEncoder: JsonEncoder[Double]) : JsonEncoder[Person] = new JsonEncoder[Person] {
override def encode(value: Person): JsonValue =
JsonObject("firstName" -> strEncoder.encode(value.firstName)
:: ("lastName" -> strEncoder.encode(value.firstName))
:: ("age" -> numberEncoder.encode(value.age) :: Nil))
}
}
Create an encode function that injects a JSON encoder instance:
import Instances._
def encode[A](in: A)(implicit jsonEncoder: JsonEncoder[A]) = jsonEncoder.encode(in)
val person = Person("name", "lastName", 25)
println(encode(person))
gives:
JsonObject(List((firstName,JsonString(name)), (lastName,JsonString(name)), (age,JsonNumber(25.0))))
Obviously you would need to create instances for each case class. To avoid that you need a function that returns a generic encoder:
def createObjectEncoder[A](fn: A => JsonObject): JsonObjectEncoder[A] =
new JsonObjectEncoder[A] {
def encode(value: A): JsonObject =
fn(value)
}
It needs a function A -> JsObject as parameter. The intuition behind this is that the compiler uses this function when traversing the HList representation of your type to create the type encoder, as it is described in the HList encoder function.
Then, you must create the HList encoder. That requires an implicit function to create the encoder for the HNil type and another for the HList itself.
implicit val hnilEncoder: JsonObjectEncoder[HNil] =
createObjectEncoder(hnil => JsonObject(Nil))
/* hlist encoder */
implicit def hlistObjectEncoder[K <: Symbol, H, T <: HList](
implicit witness: Witness.Aux[K],
hEncoder: Lazy[JsonEncoder[H]],
tEncoder: JsonObjectEncoder[T]): JsonObjectEncoder[FieldType[K, H] :: T] = {
val fieldName: String = witness.value.name
createObjectEncoder { hlist =>
val head = hEncoder.value.encode(hlist.head)
val tail = tEncoder.encode(hlist.tail)
JsonObject((fieldName, head) :: tail.fields)
}
}
The last thing that we have to do is to create an implicit function that injects an Encoder instance for a Person instance. It leverages the compiler implicit resolution to create a LabeledGeneric of your type and to create the encoder instance.
implicit def genericObjectEncoder[A, H](
implicit generic: LabelledGeneric.Aux[A, H],
hEncoder: Lazy[JsonObjectEncoder[H]]): JsonEncoder[A] =
createObjectEncoder { value => hEncoder.value.encode(generic.to(value))
}
You can code all these definitions inside the Instances object.
import Instances._
val person2 = Person2("name", "lastName", 25)
println(JsonEncoder[Person2].encode(person2))
prints:
JsonObject(List((firstName,JsonString(name)), (lastName,JsonString(lastName)), (age,JsonNumber(25.0))))
Note that you need to include in the HList encoder the Witness instance for Symbol. That allows to access the properties names at runtime. Remember that the LabeledGeneric of your Person type is something like:
String with KeyTag[Symbol with Tagged["firstName"], String] ::
Int with KeyTag[Symbol with Tagged["lastName"], Int] ::
Double with KeyTag[Symbol with Tagged["age"], Double] ::
The Lazy type it is necessary to create encoders for recursive types:
case class Person2(firstName: String, lastName: String, age: Double, person: Person)
val person2 = Person2("name", "lastName", 25, person)
prints:
JsonObject(List((firstName,JsonString(name)), (lastName,JsonString(lastName)), (age,JsonNumber(25.0)), (person,JsonObject(List((firstName,JsonString(name)), (lastName,JsonString(name)), (age,JsonNumber(25.0)))))))
Take a look to libraries like Circe or Spray-Json to see how they use Shapeless for codec derivation.
Try
implicit class StringOp(s: String) {
def :=[A](a: A): (String, A) = s -> a
}
implicit def strToJStr: String => Json.String = Json.String
implicit def dblToJNumber: Double => Json.Number = Json.Number
implicit def intToJNumber: Int => Json.Number = Json.Number(_)
sealed trait Json
object Json {
case class Object(fields: (scala.Predef.String, Json)*) extends Json
case class Array(items: List[Json]) extends Json
case class String(value: scala.Predef.String) extends Json
case class Number(value: Double) extends Json
case class Boolean(value: scala.Boolean) extends Json
case object Null extends Json
}
trait Mapper[J] {
def encode(j: J): Json
def decode(json: Json): Either[Json, J]
}
object Mapper {
implicit val `object`: Mapper[Json.Object] = ???
implicit val array: Mapper[Json.Array] = ???
implicit val stringJson: Mapper[Json.String] = ???
implicit val number: Mapper[Json.Number] = ???
implicit val boolean: Mapper[Json.Boolean] = ???
implicit val `null`: Mapper[Json.Null.type] = ???
implicit val json: Mapper[Json] = ???
implicit val int: Mapper[Int] = ???
implicit val string: Mapper[String] = ???
implicit val person: Mapper[Person] = ???
def strict[R]: IsStrict[R] =
new IsStrict[R](true)
def lenient[R]: IsStrict[R] =
new IsStrict[R](false)
class IsStrict[R](strict: Boolean) {
def has[A: Mapper](at: String): Builder[R, A :: HNil] =
new Builder(strict, at :: Nil)
}
class Builder[R, L <: HList](strict: Boolean, l: List[String]) {
def has[A: Mapper](at: String): Builder[R, A :: L] =
new Builder(strict, at :: l)
def is[L1 <: HList](decode: L1 => R)(encode: R => Json)(implicit
reverse: ops.hlist.Reverse.Aux[L, L1]): Mapper[R] = {
val l1 = l.reverse
???
}
}
}
Unfortunately this needs L1 to be explicitly specified for is
case class Person(firstName: String, lastName: String, age: Int)
Mapper
.strict[Person]
.has[String]("firstName")
.has[String]("lastName")
.has[Int]("age")
.is[String :: String :: Int :: HNil] {
case (firstName :: lastName :: age :: HNil) =>
new Person(firstName, lastName, age)
} { person =>
Json.Object(
"firstName" := person.firstName,
"lastName" := person.lastName,
"age" := person.age
)
}
otherwise it's Error: missing parameter type for expanded function.
The argument types of an anonymous function must be fully known.
One way to improve inference is to move implicit reverse to class Builder but this is less efficient: an HList will be reversed in every step , not only in the last step.
Another way is to introduce helper class
def is(implicit reverse: ops.hlist.Reverse[L]) = new IsHelper[reverse.Out]
class IsHelper[L1 <: HList]{
def apply(decode: L1 => R)(encode: R => Json): Mapper[R] = {
val l1 = l.reverse
???
}
}
but then apply (or other method name) should be explicit
Mapper
.strict[Person]
.has[String]("firstName")
.has[String]("lastName")
.has[Int]("age")
.is.apply {
case (firstName :: lastName :: age :: HNil) =>
new Person(firstName, lastName, age)
} { person =>
Json.Object(
"firstName" := person.firstName,
"lastName" := person.lastName,
"age" := person.age
)
}
otherwise compiler mistreats decode as reverse.
I am using the Play Framework and trying to build JSON validator for a class with abstract members. Shown below, the DataSource class is the base class which I am trying to validate the format against.
// SourceTypeConfig Trait.
trait SourceTypeConfig
final case class RDBMSConfig(...) extends SourceTypeConfig
object RDBMSConfig { implicit val fmt = Json.format[RDBMSConfig] }
final case class DirectoryConfig(
path: String,
pathType: String // Local, gcloud, azure, aws, etc.
) extends SourceTypeConfig
object DirectoryConfig { implicit val fmt = Json.format[DirectoryConfig] }
// FormatConfig trait.
trait FormatConfig
final case class SQLConfig(...) extends FormatConfig
object SQLConfig { implicit val fmt = Json.format[SQLConfig]}
final case class CSVConfig(
header: String,
inferSchema: String,
delimiter: String
) extends FormatConfig
object CSVConfig { implicit val fmt = Json.format[CSVConfig]}
// DataSource base class.
case class DataSource(
name: String,
sourceType: String,
sourceTypeConfig: SourceTypeConfig,
format: String,
formatConfig: FormatConfig
)
What I am hoping to accomplish:
val input: JsValue = Json.parse(
"""
{
"name" : "test1",
"sourceType" : "directory",
"sourceTypeConfig" : {"path" : "gs://test/path", "pathType" "google"},
"format" : "csv",
"formatConfig" : {"header" : "yes", "inferSchema" : "yes", "delimiter" : "|"}
}
"""
)
val inputResult = input.validate[DataSource]
What I am struggling with is building the DataSource object and defining its reads/writes/format. I would like it to contain a match based on the sourceType and format values that direct it to point towards the associated sourceTypeConfig and formatConfig's formats so it can parse out the JSON.
Instead of building a parser at the DataSource level, I defined parsers at the SourceConfig and FormatConfig levels, similar to what is shown below.
sealed trait SourceConfig{val sourceType: String}
object SourceConfig{
implicit val fmt = new Format[SourceConfig] {
def reads(json: JsValue): JsResult[SourceConfig] = {
def from(sourceType: String, data: JsObject): JsResult[SourceConfig] = sourceType match {
case "RDBMS" => Json.fromJson[RDBMSConfig](data)(RDBMSConfig.fmt)
case "directory" => Json.fromJson[DirectoryConfig](data)(DirectoryConfig.fmt)
case _ => JsError(s"Unknown source type: '$sourceType'")
}
for {
sourceType <- (json \ "sourceType").validate[String]
data <- json.validate[JsObject]
result <- from(sourceType, data)
} yield result
}
def writes(source: SourceConfig): JsValue =
source match {
case b: RDBMSConfig => Json.toJson(b)(RDBMSConfig.fmt)
case b: DirectoryConfig => Json.toJson(b)(DirectoryConfig.fmt)
}
}
}
Then, DataSource could be simply defined as:
object DataSource { implicit val fmt = Json.format[DataSource] }
Another option is to use play-json-derived-codecs library:
libraryDependencies += "org.julienrf" %% "play-json-derived-codecs" % "4.0.0"
import julienrf.json.derived.flat
implicit val format1: OFormat[RDBMSConfig] = Json.format[RDBMSConfig]
implicit val format2: OFormat[DirectoryConfig] = Json.format[DirectoryConfig]
implicit val format3: OFormat[SourceTypeConfig] = flat.oformat((__ \ "sourceType").format[String])
I am not much familier with spray json, but I have to convert the below json into Array[myTest]
Below is the code, but it doesnt work. It throws the following errors: How do I fix them?
Error:(19, 54) Cannot find JsonReader or JsonFormat type class for Array[A$A61.this.myTest]
lazy val converted= trainingDataRef.toJson.convertTo[Array[myTest]]
^
Error:(19, 54) not enough arguments for method convertTo: (implicit evidence$1: spray.json.JsonReader[Array[A$A61.this.myTest]])Array[A$A61.this.myTest].
Unspecified value parameter evidence$1.
lazy val converted= trainingDataRef.toJson.convertTo[Array[myTest]]
^
Error:(10, 61) could not find implicit value for evidence parameter of type spray.json.DefaultJsonProtocol.JF[Map[String,Any]]
implicit val format: RootJsonFormat[myTest] = jsonFormat3(myTest.apply)
^
Code: ^
import spray.json.DefaultJsonProtocol._
import spray.json._
case class myTest (
id: String,
classDetails: Map[String, Any],
school: Map[String, Any])
object myTest {
implicit val format: RootJsonFormat[myTest] = jsonFormat3(myTest.apply)
}
val trainingDataRef = """[{"id":"my-id","classDetails":{"sec":"2","teacher":"John"},"school":{"name":"newschool"}}]"""
println(trainingDataRef.getClass)
val converted= trainingDataRef.toJson.convertTo[Array[myTest]]
println(converted)
spray-json has good documentation, try take a look there. Basically, you have to define your case classes and implement JsonFormat for them:
import spray.json.DefaultJsonProtocol._
import spray.json._
case class ClassDetails(sec: String, teacher: String)
object ClassDetails {
implicit val format: RootJsonFormat[ClassDetails] = jsonFormat2(ClassDetails.apply)
}
case class School(name: String)
object School {
implicit val format: RootJsonFormat[School] = jsonFormat1(School.apply)
}
case class ClassInfo
(
id: String,
classDetails: ClassDetails,
school: School
)
object ClassInfo {
implicit object ClassInfoFormat extends RootJsonFormat[ClassInfo] {
def write(c: ClassInfo): JsValue = JsObject(
"id" -> JsString(c.id),
"classDetails" -> c.classDetails.toJson,
"school" -> c.school.toJson
)
def read(value: JsValue): ClassInfo = {
value.asJsObject.getFields("id", "classDetails", "school") match {
case Seq(JsString(name), details, school) =>
new ClassInfo(name, details.convertTo[ClassDetails], school.convertTo[School])
case _ => throw new DeserializationException("ClassInfo expected")
}
}
}
}
val json = """[{"id":"my-id","classDetails":{"sec":"2","teacher":"John"},"school":{"name":"newschool"}}]"""
// JSON string to case classes
val classInfos = json.parseJson.convertTo[Seq[ClassInfo]]
classInfos.zipWithIndex.foreach { case (c, idx) =>
println(s"$idx => $c")
}
println
// Seq[ClassInfo] to JSON
println(s"$classInfos: ")
println(classInfos.toJson.prettyPrint)
I have domain model like:
sealed trait MyTrait
case class MyObject(a: String) extends MyTrait
case class MyArray(a: Seq(MyObject)) extends MyTrait
Example usages would look like:
// array
[{"a": "foo"}, {"a": "bar"}]
//object
{"a": "foobar"}
I want to write one Spray JSON reader for MyTrait like:
implicit val myTraitReader: RooJsonReader[MyTrait] = new RootJsonReader[MyTrait] {
override def read(json: JsValue): MyTrait = {
// MAGIC?
// I need to be able to distinguish between JsObject and JsArray
// If I do json.asJsObject this is a JsObject then :D
// there is not json.asJsArray?
// How to pattern match on this use case?
}
}
All questions are commented out. :D
Thanks!
How about something like below?
implicit val myTraitReader: RootJsonReader[MyTrait] = new RootJsonReader[MyTrait] {
import MyObjectProtocol._
import MyArrayProtocol._
override def read(json: JsValue): MyTrait = json match {
case jsObj: JsObject => ??? // Convert to MyObject here
case jsArr: JsArray => ??? // Convert to MyArray here
case _ => ??? // A JsNull here??
}
}
I use Scala case classes and Play "format" to validate JSON messages. For example:
case class Person(name: String)
implicit val formatPerson = Json.format[Person]
The following JSON:
{
"name": "Alice"
}
would be validated through the method
Json.validate[Person](json)
Now I would like to validate a JSON message with a field "x" that could be a String or a Integer.
For example the two following messages would be both validated with the same method:
{
"x": "hello"
}
{
"x": 8
}
I tried with the following trick but it does not work:
case class Foo(x: Either[String,Int])
implicit val formatFoo = Json.format[Foo]
When I try to define the format for Foo class, the compiler says: "No apply function found matching unapply parameters". Thanks in advance.
You'll have to define a custom Format[Foo]. This should work for Play 2.4
implicit val formatFoo = new Format[Foo]{
override def writes(o: Foo): JsValue = o.x match {
case Left(str) => Json.obj("x" -> str)
case Right(n) => Json.obj("x" -> n)
}
override def reads(json: JsValue): JsResult[Foo] = json \ "x" match {
case JsDefined(JsString(str)) => JsSuccess(Foo(Left(str)))
case JsDefined(JsNumber(n)) if n.isValidInt => JsSuccess(Foo(Right(n.toInt)))
case _ => JsError("error")
}
}