How can I programmatically create a validation contract at compile-time? - json

I apologize in advance if this is an XY problem.
tl;dr:
I'd like to have a compile-time map of type [Request.type, Response.type] so I can effectively say if I send message Request, a CLI should, at compile-time, know how to deserialize its expected Response, irrespective of the fact that it won't know what type of request is sent until runtime.
too long; still read:
I have a CLI which communicates with an HTTP server and depending on the type of message sent to the HTTP server, I'd like to validate the JSON response against a case case.
For instance, if I send the HTTP server an AddFoo message, I might want to validate that the JSON response can be deserialized into an AddedFoo, etc.
My current solution is quite hacky. Using play-json, I'm attempting to parse the JSON response using a mapping from config.mode (i.e., command issued to the CLI) to the expected responses' implicit Reads.
My code looks something like this:
val modeToResponseReads: Map[String, Reads[_]] = Map(
Modes.ADD_FOO -> AddedFoo.addedFooReads,
Modes.ADD_BOO -> AddedBoo.addedBooReads,
Modes.GET_WOO -> GetWooResponse.getWooReads,
)
parser.parse(args, MyConfig()) match {
case Some(config) => try {
val exec = new MyHttpExecutor(remoteUri, config)
val res = Await.result(exec.getResponse, 100.seconds)
// passing `Reads` to `as` because JsValue#as[T] cannot be
// applied at runtime -- only compile-time.
val _ = Json.parse(res.json.toString)
.as(modeToResponseReads(config.mode))
exec.actorSystem.terminate()
exec.wsClient.close()
} catch {
case t: Throwable => logger.error(t.getMessage)
}
case None => {
logger.error("Bad arguments.")
sys.exit(1)
}
}
While this works, it's an incredible kludge that becomes increasingly unmaintainable with an increasing number of messages. Further, I've found that this pattern will need to be replicated anywhere some type of validate or conversion will need to happen (e.g., Future[Any] being converted to Future[AddedFoo]).
Surely my approach isn't the right way... how is this traditionally done? If it is the right way (please no), are there optimizations that can be made?

I managed to accomplish this by encoding the contract directly into the child Request classes. Namely, the child Request classes would hold a ResponseType type with the base class enforcing the covariant type.
So I can do something like this:
abstract class Response
abstract class Request[+A <: Response]
case class Foo(id: String)
object Foo {
implicit val fooReads = Json.reads[Foo]
implicit val fooFormat = Json.format[Foo]
}
case class FooResponse(foo: Foo) extends Response {
def greet = println("woo hoo!")
}
object FooResponse {
implicit val fooRespReads = Json.reads[FooResponse]
implicit val fooRespFormat = Json.format[FooResponse]
}
case class FooRequest() extends Request[FooResponse] {
type ResponseType = FooResponse
}
object Main extends App {
val req: FooRequest = new FooRequest()
val foo = Foo("12345")
val resp = new FooResponse(foo)
val respJsonString = Json.toJson(resp).toString
println(Json.parse(respJsonString).as[req.ResponseType])
}

Related

How can I parse an object of a bounded type T from JSON from an implicit request?

I have some simple messages with implicit Json.reads and Json.formats defined in their companion objects. All of these messages extend MyBaseMessage.
In other words, for any T <: MyBaseMessage, T is (de)serializable.
These messages represent simple CRUD operations to be performed on a cluster, so there's an Play server that will sit between a CLI sending JSON and the Cluster. Because the operations are simple, I should be able to make some very generic Actions on the Play side: when I receive JSON at an endpoint, deserialize the message according to the endpoint and forward that message to the cluster.
My ultimate goal is to do something like this:
// AddBooMessage extends MyBaseMessage
def addBoo = FooAction[AddBooMessage]
// AddMooMessage extends MyBaseMessage
def addMoo = FooAction[AddMooMessage]
// etc. ...
So when a request is sent to the route corresponding to the addBoo message, the request's JSON will be parsed into an AddBooMessage message and pushed to the cluster. Repeat ad nauseam.
I have the following written:
private def FooAction[T <: MyBaseMessage] = Action {
implicit request =>
parseAndForward[T](request)
}
private def parseAndForward[T <: MyBaseMessage](request: Request[AnyContent]) = {
val parsedRequest = Json.parse(request.body.toString).as[T]
Logger.info(s"Got '$parsedRequest' request. Forwarding it to the Cluster.")
sendToCluster(parsedRequest)
}
But I find the following error:
No Json deserializer found for type T. Try to implement an implicit Reads or Format for this type.
However, all of these messages are serializable and have both Reads and Format defined for them.
I tried passing (implicit fjs: Reads[T]) to parseAndForward in hopes to implicitly provide the Reads required (though it should already be implicitly provided), but it didn't help.
How can I solve this problem?
JsValue#as[A] needs an implicit Reads[A] in order to deserialize JSON to some type A. That is, the error message you're getting is caused because the compiler can't guarantee there is a Reads[T] for any type T <: MyBaseMessage. Assuming sendToCluster is parameterized the same way, this can easily by fixed by simply requiring an implicit Reads[T] in each method call. It sounds like you were close, and just needed to take things a step further by requiring the Reads[T] from FooAction, as well (since that call is where the type is determined).
private def FooAction[T <: MyBaseMessage : Reads] = Action { implicit request =>
parseAndForward[T](request)
}
private def parseAndForward[T <: MyBaseMessage : Reads](request: Request[AnyContent]) = {
val parsedRequest = Json.parse(request.body.toString).as[T]
Logger.info(s"Got '$parsedRequest' request. Forwarding it to the Cluster.")
sendToCluster(parsedRequest) // Assuming this returns a `Future[Result]`
}
If your intention if you use the above code by manually supplying the type parameter, this will work just fine.
There are some other improvements I think you can make here. First, if you always expect JSON, you should require the parse.json BodyParser. This will return a BadRequest if what is received isn't even JSON. Second, as will throw an exception if the received JSON cannot be deserialized into the expected type, you can use JsValue#validate to do this more safely and fold the result to handle the success and error cases explicitly. For example:
private def FooAction[T <: MyBaseMessage] = Action.async(parse.json) { implicit request =>
parseAndForward[T](request)
}
private def parseAndForward[T <: MyBaseMessage](request: Request[JsValue]) = {
request.body.validate[T].fold(
error => {
Logger.error(s"Error parsing request: $request")
Future.successful(BadRequest)
},
parsed => {
Logger.info(s"Got '$parsed' request. Forwarding it to the Cluster.")
sendToCluster(parsed)
}
)
}

Json "Validate" for Play

For the validate method on request.body it matches the attribute name and value type of the json object to those defined in the model definition. Now if I were to add an extra attribute to the json object and try to validate it, it passes as a JsSuccess when it shouldn't.
{
"Name": "Bob",
"Age": 20,
"Random_Field_Not_Defined_in_Models": "Test"
}
My Person Class is defined as follows
case class Person(name: String, age: Int)
I'm assuming you've been using the built-in Reads[T] or Format[T] converters that Play gives you via Json.reads[T], e.g.:
import play.api.libs.json._
val standardReads = Json.reads[Person]
While these are super-handy, if you need additional validation, you'll have to define a custom Reads[Person] class; but fortunately we can still leverage the built-in JSON-to-case-class macro to do the basic checking and conversion, and then add an extra layer of custom checks if things seem OK:
val standardReads = Json.reads[Person]
val strictReads = new Reads[Person] {
val expectedKeys = Set("name", "age")
def reads(jsv:JsValue):JsResult[Person] = {
standardReads.reads(jsv).flatMap { person =>
checkUnwantedKeys(jsv, person)
}
}
private def checkUnwantedKeys(jsv:JsValue, p:Person):JsResult[Person] = {
val obj = jsv.asInstanceOf[JsObject]
val keys = obj.keys
val unwanted = keys.diff(expectedKeys)
if (unwanted.isEmpty) {
JsSuccess(p)
} else {
JsError(s"Keys: ${unwanted.mkString(",")} found in the incoming JSON")
}
}
}
Note how we utilize standardReads first, to make sure we're dealing with something that can be converted to a Person. No need to reinvent the wheel here.
We use flatMap to effectively short-circuit the conversion if we get a JsError from standardReads - i.e. we only call checkUnwantedKeys if needed.
checkUnwantedKeys just uses the fact that a JsObject is really just a wrapper around a Map, so we can easily check the names of the keys against a whitelist.
Note that you could also write that flatMap using a for-comprehension, which starts to look a lot cleaner if you need even more checking stages:
for {
p <- standardReads.reads(jsv)
r1 <- checkUnexpectedFields(jsv, p)
r2 <- checkSomeOtherStuff(jsv, r1)
r3 <- checkEvenMoreStuff(jsv, r2)
} yield r3
If you want to avoid too much boilerplate it is possible to make a more generic solution using a little bit of scala reflection:
import play.api.libs.json._
import scala.reflect.runtime.universe._
def checkedReads[T](underlyingReads: Reads[T])(implicit typeTag: TypeTag[T]): Reads[T] = new Reads[T] {
def classFields[T: TypeTag]: Set[String] = typeOf[T].members.collect {
case m: MethodSymbol if m.isCaseAccessor => m.name.decodedName.toString
}.toSet
def reads(json: JsValue): JsResult[T] = {
val caseClassFields = classFields[T]
json match {
case JsObject(fields) if (fields.keySet -- caseClassFields).nonEmpty =>
JsError(s"Unexpected fields provided: ${(fields.keySet -- caseClassFields).mkString(", ")}")
case _ => underlyingReads.reads(json)
}
}
}
Then you can specify your reads instances as:
implicit val reads = checkedReads(Json.reads[Person])
This leverages a fair bit of Scala type magic and also the reflection library (that lets you look at fields on classes).
Rather than relying on a fixed set of fields the classFields method gets all of the fields dynamically for the case class (type param T). It looks at all of the members and collects only the case class accessors (otherwise we'd pick up methods like toString). It returns a Set[String] of field names.
You'll notice that the checkedReads takes an implicit TypeTag[T]. This is supplied by the compiler at compile time and used by the typeOf method.
The remaining code is fairly self explanatory. If the incoming json matches our first case (it is a JsObject and there are fields not on the case class) then we return a JsError. Otherwise we pass it on to the underlying reader.

How to fail on json with unused fields in play-json scala? [duplicate]

For the validate method on request.body it matches the attribute name and value type of the json object to those defined in the model definition. Now if I were to add an extra attribute to the json object and try to validate it, it passes as a JsSuccess when it shouldn't.
{
"Name": "Bob",
"Age": 20,
"Random_Field_Not_Defined_in_Models": "Test"
}
My Person Class is defined as follows
case class Person(name: String, age: Int)
I'm assuming you've been using the built-in Reads[T] or Format[T] converters that Play gives you via Json.reads[T], e.g.:
import play.api.libs.json._
val standardReads = Json.reads[Person]
While these are super-handy, if you need additional validation, you'll have to define a custom Reads[Person] class; but fortunately we can still leverage the built-in JSON-to-case-class macro to do the basic checking and conversion, and then add an extra layer of custom checks if things seem OK:
val standardReads = Json.reads[Person]
val strictReads = new Reads[Person] {
val expectedKeys = Set("name", "age")
def reads(jsv:JsValue):JsResult[Person] = {
standardReads.reads(jsv).flatMap { person =>
checkUnwantedKeys(jsv, person)
}
}
private def checkUnwantedKeys(jsv:JsValue, p:Person):JsResult[Person] = {
val obj = jsv.asInstanceOf[JsObject]
val keys = obj.keys
val unwanted = keys.diff(expectedKeys)
if (unwanted.isEmpty) {
JsSuccess(p)
} else {
JsError(s"Keys: ${unwanted.mkString(",")} found in the incoming JSON")
}
}
}
Note how we utilize standardReads first, to make sure we're dealing with something that can be converted to a Person. No need to reinvent the wheel here.
We use flatMap to effectively short-circuit the conversion if we get a JsError from standardReads - i.e. we only call checkUnwantedKeys if needed.
checkUnwantedKeys just uses the fact that a JsObject is really just a wrapper around a Map, so we can easily check the names of the keys against a whitelist.
Note that you could also write that flatMap using a for-comprehension, which starts to look a lot cleaner if you need even more checking stages:
for {
p <- standardReads.reads(jsv)
r1 <- checkUnexpectedFields(jsv, p)
r2 <- checkSomeOtherStuff(jsv, r1)
r3 <- checkEvenMoreStuff(jsv, r2)
} yield r3
If you want to avoid too much boilerplate it is possible to make a more generic solution using a little bit of scala reflection:
import play.api.libs.json._
import scala.reflect.runtime.universe._
def checkedReads[T](underlyingReads: Reads[T])(implicit typeTag: TypeTag[T]): Reads[T] = new Reads[T] {
def classFields[T: TypeTag]: Set[String] = typeOf[T].members.collect {
case m: MethodSymbol if m.isCaseAccessor => m.name.decodedName.toString
}.toSet
def reads(json: JsValue): JsResult[T] = {
val caseClassFields = classFields[T]
json match {
case JsObject(fields) if (fields.keySet -- caseClassFields).nonEmpty =>
JsError(s"Unexpected fields provided: ${(fields.keySet -- caseClassFields).mkString(", ")}")
case _ => underlyingReads.reads(json)
}
}
}
Then you can specify your reads instances as:
implicit val reads = checkedReads(Json.reads[Person])
This leverages a fair bit of Scala type magic and also the reflection library (that lets you look at fields on classes).
Rather than relying on a fixed set of fields the classFields method gets all of the fields dynamically for the case class (type param T). It looks at all of the members and collects only the case class accessors (otherwise we'd pick up methods like toString). It returns a Set[String] of field names.
You'll notice that the checkedReads takes an implicit TypeTag[T]. This is supplied by the compiler at compile time and used by the typeOf method.
The remaining code is fairly self explanatory. If the incoming json matches our first case (it is a JsObject and there are fields not on the case class) then we return a JsError. Otherwise we pass it on to the underlying reader.

How to setup implicit json convertion for spray+akka actor

I'm Java developer and pretty new to scala.
I'm implementing some rest API that use spray and akka
The API should expose some kind of user CRUD. I'll use only create user in this question...
trait DefaultJsonFormats extends DefaultJsonProtocol with SprayJsonSupport with MetaMarshallers {}
class RegistrationService(registration: ActorRef)
(implicit executionContext: ExecutionContext)
extends Directives with DefaultJsonFormats {
implicit val timeout = Timeout(2.seconds)
implicit val userFormat = jsonFormat3(User)
implicit val registerFormat = jsonFormat1(Register)
implicit val registeredFormat = jsonFormat1(Registered)
val route =
path("register") {
post { handleWith { ru: Register => (registration ? ru).mapTo[Registered] } }
}
//------ Actor
object RegistrationActor {
case class User(id:String, name:String)
case class Register(user: User)
case class Registered(status: String)
case object NotRegistered
}
class RegistrationActor(implDef: String) extends Actor {
def receive: Receive = {
case Register(user)=>
val status=// create user real code with return status
sender ! new Registered(status)
} }
In this approach the json serialization and desiarelization is pretty annoying. For every object I need to deal with API I must define the appropriate format
implicit val userFormat = jsonFormat3(User)
implicit val registerFormat = jsonFormat1(Register)
implicit val registeredFormat = jsonFormat1(Registered)
I would like to avoid such definition and use some general json converter and return a pojo objects, so the conversion will happen under-the-hood
The question is how can I change this code to use by default Gson/Jackson/Spray default converter and avoid definition of the implicit ... jsonFormats?
For every object I need to deal with API I must define the appropriate format
It is normal to do this once, in a "JsonProtocol" class and import that where needed, rather than defining new formats each time:
import MyJsonProtocol._
val route =
path("register") {
post { handleWith { ru: Register => (registration ? ru).mapTo[Registered] } }
how can I change this code to use by default Gson/Jackson/Spray default converter and avoid definition of the implicit ... jsonFormats?
You would need to declare an implicit marshaller from Registered to HttpResponse (or an intermediate value like String) which was backed by Jackson instead of spray-json, then import that marshaller instead of SprayJsonSupport.
Have a look at the implementation of SprayJsonSupport to see how to do this. It's fairly straightforward, if you're comfortable with implicit conversions.
You can also see how this is done in Json4sSupport in Spray -- that trait implements a Marshaller[T, String] for ALL types T. Then, at runtime, the Json4s library will try to serialize the object to JSON.
In this approach the json serialization and desiarelization is pretty annoying
There are two main advantages of spray-jsons approach over Jackson's:
There is no reflection, so it is faster at runtime
This is no runtime determining of JSON formats, so any issues are caught at compile-time

Handling Pk[Int] values in spray-json

[edit]
So, i got a quick and dirty solution, thanks to Edmondo1984, I don't know if it's the best solution. I don't handle null values with pattern matching at the write function. You can read more details about my problem after this editing. Here is my code now:
object DBNames extends DefaultJsonProtocol {
implicit val pkFormat: JsonFormat[Pk[Int]] = new JsonFormat[Pk[Int]] {
def write(obj: Pk[Int]): JsValue = JsNumber(obj.get)
def read(json: JsValue): Pk[Int] = json.asJsObject.getFields("id") match {
case Seq(JsNumber(id)) => new Pk[Int] { id.toInt }
case _ => throw new DeserializationException("Int expected")
}
}
implicit val nameFormat = jsonFormat2(Name)
jsonFormat2 will implicitly use pkFormat to parse Pk[Int] values.
In my controller class I have this:
def listNames() = Action {
val names = DBNames.findAll()
implicit val writer = DBNames.nameFormat
var json = names.toJson
Ok(json.toString()).as("application/json")
}
I had to get the nameFormat from my model and make it implicit, so bars.toJson could use it to parse the Seq[Name] names.
[/edit]
I'm trying to use Play! Framework with Scala, I'm new to Scala programming and Play Framework, and everything seems nice, but I'm working on this problem during several hours and didn't find a solution.
I have a Case Class:
case class Name (id: Pk[Int], name: String)
And an object to deal with MySql. I created a implicit val nameFormat = jsonFormat2(Name) to deal with JSON.
object DBNames extends DefaultJsonProtocol {
implicit val nameFormat = jsonFormat2(Name)
var parser =
{
get[Pk[Int]]("id") ~
get[String]("name") map {
case id ~ name => Name(id,name)
}
}
def findAll():Seq[Name] =
{
DB.withConnection {
implicit connection =>
SQL("select * from names").as(DBNames.parser *)
}
}
def create(name: Name){
DB.withConnection {
implicit connection =>
SQL("insert into names (name) values ({name})").on(
'name -> name.name
).executeUpdate()
}
}
}
But when I try to compile it, Play! gives me this result:
[error] D:\ProjetosJVM\TaskList\app\models\Names.scala:20: could not find implicit value for evidence parameter of type models.DBNames.JF[anorm.Pk[Int]]
It seems like he couldn't find a way to parse the id value, since it is a Pk[Int] value.
So, by reading this: https://github.com/spray/spray-json I didn't found a way to parse it without creating a complete object parser like they show in the documentation:
object MyJsonProtocol extends DefaultJsonProtocol {
implicit object ColorJsonFormat extends RootJsonFormat[Color] {
def write(c: Color) = JsObject(
"name" -> JsString(c.name),
"red" -> JsNumber(c.red),
"green" -> JsNumber(c.green),
"blue" -> JsNumber(c.blue)
)
def read(value: JsValue) = {
value.asJsObject.getFields("name", "red", "green", "blue") match {
case Seq(JsString(name), JsNumber(red), JsNumber(green), JsNumber(blue)) =>
new Color(name, red.toInt, green.toInt, blue.toInt)
case _ => throw new DeserializationException("Color expected")
}
}
}
}
I have a "big" (actually small) project where I want to make most of things work with Ajax, so I think this is not a good way to do it.
How can I deal with JSON objects in this project, where almost all case classes will have a "JSON parser", without creating large ammounts of code, like the snippet above? And also, how can I make it work with an Seq[Name]?
You don't need to write a complete parser. The compiler says:
[error] D:\ProjetosJVM\TaskList\app\models\Names.scala:20: could not find implicit
value for evidence parameter of type models.DBNames.JF[anorm.Pk[Int]]
The scala compiler is looking for an implicit parameter of type JF[anorm.Pk[Int]] and there is no such an implicit parameter in scope. What is JF[anorm.Pk[Int]]? Well, you need to know the library and I didn't, so I had browsed spray-json source and found out:
trait StandardFormats {
this: AdditionalFormats =>
private[json] type JF[T] = JsonFormat[T] // simple alias for reduced verbosity
so JF[T] is just an alias for JsonFormat[T]. It all make sense: PK[Int] is a class coming from Anorm and spray-json provides out-of-the-box json support for standard types, but does not even know Anorm exists So you have to code your support for Pk[Int] and make it implicit in scope.
You will have code like the following:
object DBNames extends DefaultJsonProtocol {
implicit val pkFormat : JsonFormat[Pk[Int]] = new JsonFormat[Pk[Int]] {
//implementation
}
// rest of your code
}
If you have just started with Scala, you would probably have to read more about implicits and their resolution. I am providing you with a minimal answer: once you have provided the right implementation, your code will compile. I suggest you to refer to the javadoc of anorm.Pk and of JsonFormat to understand how to implement it correctly for your type.
Pk looks like scala.Option and in StandardFormats source code inside spray-json you find the JsonFormat implementation for Option, from which you can copy