I have some models in a Play! application that I would like to serialize/deserialize to and from JSON. I used to have separate methods for that, but I have seen that the preferred way is to give an implicit instance of Formats[T] or Reads[T], like
import play.api.libs.json.{ JsValue, Reads }
case class Foo(bar: Int, ...)
object Foo {
implicit object FooReads extends Reads[Foo] {
def reads(json: JsValue): Foo = //whatever
}
}
Now, it may happen that the model has the correct fields in the JSON, but it does not validate. In this case, I am not able to deserialize - I should get an exception when using json.as[Foo] or None when using json.asOpt[Foo].
If I throw an exception when I find a field that does not validate, everything seems to work as expected. But I took the care of trying to find out what exception I should throw, and in the source for JsValue I found this
def asOpt[T](implicit fjs: Reads[T]): Option[T] = fjs.reads(this).fold(
valid = v => Some(v),
invalid = _ => None
).filter {
case JsUndefined(_) => false
case _ => true
}
Now, I cannot understand how this is supposed to work. The implicit instance of fjs is supplied by myself in the companion object, so I know that fjs.reads(this) either returns a Foo or throws an exception.
Where is this fold coming from? It certainly is not a method on Foo. I guess one could have an implicit conversion, but it should be from Any to something with a fold method, so it could not be of much interest. Even worse, if fjs.reads(this) throws an exception, there is nothing to catch it!
So, how should one handle invalid input in the JSON in an instance of Reads[T]? And how does the mechanism above actually work?
Looking at JsonValue.scala in Play 2.0.x:
def asOpt[T](implicit fjs: Reads[T]): Option[T] = catching(classOf[RuntimeException]).opt(fjs.reads(this)).filter {
case JsUndefined(_) => false
case _ => true
}
In fact the code is using scala.util.control.Exception.catching[T](exceptions: Class[_]*): Catch[T], which returns a Catch[T]. Then it calls opt(...) on it. If an exception is thrown, then it will return a None instead of an instance of T.
So, when you get an error deserializing, you can safely throw an exception.
Related
My question concerns the second solution offered by mixel here: Scala Circe with generics
Note that the trait named Auto in Circe has been renamed to AutoDerivation in the current version of Circe.
I am using the solution mixel provides in his StackOverflow solution but have not been able to get it to work. I have tried things like updating my Circe version to the most recent one and making sure the Macro Paradise plugin is imported, but still no luck.
Here is my code. The first is its own file, called CirceGeneric.
import io.circe._
import io.circe.parser._
import io.circe.generic.extras._
object CirceGeneric {
trait JsonEncoder[T] {
def apply(in: T): Json
}
trait JsonDecoder[T] {
def apply(s: String): Either[Error, T]
}
object CirceEncoderProvider {
def apply[T: Encoder]: JsonEncoder[T] = new JsonEncoder[T] {
def apply(in: T) = Encoder[T].apply(in)
}
}
object CirceDecoderProvider {
def apply[T: Decoder]: JsonDecoder[T] = new JsonDecoder[T] {
def apply(s: String) = decode[T](s)
}
}
}
object Generic extends AutoDerivation {
import CirceGeneric._
implicit def encoder[T: Encoder]: JsonEncoder[T] = CirceEncoderProvider[T]
implicit def decoder[T: Decoder]: JsonDecoder[T] = CirceDecoderProvider[T]
}
The second is a method for unit testing that uses the Akka function responseAs. The method appears in a class called BaseServiceTest.
def responseTo[T]: T = {
def response(s: String)(implicit d: JsonDecoder[T]) = {
d.apply(responseAs[String]) match {
case Right(value) => value
case Left(error) => throw new IllegalArgumentException(error.fillInStackTrace)
}
}
response(responseAs[String])
}
The idea is to convert the result of responseAs[String] (which returns a string) into a decoded case class.
The code is not behaving as expected. Intellij does not detect any missing implicits, but when compilation time comes around, I am getting problems. I should mention that the BaseServiceTest file contains imports for CirceGeneric._ and Generic._, so a missing import statement is not the problem.
[error] [...]/BaseServiceTest.scala:59: could not find implicit value for parameter d: [...]CirceGeneric.JsonDecoder[T]
[error] response(responseAs[String])
Either the implicit conversion from Decoder[T] to JsonDecoder[T] is not happening, or the Decoder[T] instance is not being created. Either way, something is wrong.
You still need a Decoder or JsonDecoder context bound on responseTo.
def responseTo[T : Decoder]: T = ...
This is because all your code, and indeed mixel's code in the linked answer, is about abstracting from a Decoder out to a JsonDecoder trait which can be used for cross-library support. But you still don't have any way of constructing one without an underlying Decoder instance.
Now, there are some ways of automatically generating Decoders for (for instance) case classes contained in circe.generics.auto, but at this point in your code
def responseTo[T]: T = {
def response(s: String)(implicit d: JsonDecoder[T]) = ...
...
}
you're asking the compiler to be able to provide an implicit JsonDecoder (i.e., in your setup, Decoder) instance for any arbitrary type. As the accepted answer to the linked question explains, that's not possible.
You need to delay the implicit resolution to the point where you know what type you're dealing with - in particular, that you can provide a Decoder[T] instance for it.
EDIT: In your response to your comment regarding what the point is if you can't create JsonDecoders for all types...
My understanding of the linked question is that they're trying to abstract away the circe library in order to allow swapping out the JSON library implementation. This is done as follows:
add the JsonDecoder type class
have a package json which contains implicits (using Circe) for constructing them automatically via the package object extending AutoDerivation
have external code only refer to JsonDecoder and import the implicits in the json package
Then all the JSON serialization and implicit resolution works out without ever needing the calling code to reference io.circe, and it's easy to switch over the json/JsonDecoder to another JSON library if desired. But you're still going to have to use the JsonDecoder context bound, and be restricted to working with types where such an implicit can be constructed. Which is not every type.
No instance of play.api.libs.json.Format is available for models.AccountStatus in the implicit scope.
This is the code taken from a github page, and only class names and variable names are changed.
package models
import slick.jdbc.H2Profile._
import play.api.libs.json._
case class Account(id: Long, name: String, category: Int, status:AccountStatus)
object Account {
implicit val accountFormat = Json.format[Account]
}
sealed abstract class AccountStatus(val as:Int)
object AccountStatus{
final case object Draft extends AccountStatus(0)
final case object Active extends AccountStatus(1)
final case object Blocked extends AccountStatus(2)
final case object Defaulter extends AccountStatus(3)
implicit val columnType: BaseColumnType[AccountStatus] = MappedColumnType.base[AccountStatus,Int](AccountStatus.toInt, AccountStatus.fromInt)
private def toInt(as:AccountStatus):Int = as match {
case Draft => 0
case Active => 1
case Blocked => 2
case Defaulter => 3
}
private def fromInt(as: Int): AccountStatus = as match {
case 0 => Draft
case 1 => Active
case 2 => Blocked
case 3 => Defaulter
_ => sys.error("Out of bound AccountStatus Value.")
}
}
https://github.com/playframework/play-scala-slick-example/blob/2.6.x/app/models/Person.scala
So, this code needs to be added inside of the object AccountStatus code block since we need to use fromInt to transform an Int to an AccountStatus. This is a Reads defined for AccountStatus:
implicit object AccountStatusReads extends Reads[AccountStatus] {
def reads(jsValue: JsValue): JsResult[AccountStatus] = {
(jsValue \ "as").validate[Int].map(fromInt)
}
}
What's a Reads? It's just a trait that defines how a JsValue (the play class encapsulating JSON values) should be deserialized from JSON to some type. The trait only requires one method to be implemented, a reads method which takes in some json and returns a JsResult of some type. So you can see in the above code that we have a Reads that will look for a field in JSON called as and try to read it as an integer. From there, it will then transform it into an AccountStatus using the already defined fromInt method. So for example in the scala console you could do this:
import play.api.libs.json._
// import wherever account status is and the above reader
scala> Json.parse("""{"as":1}""").as[AccountStatus]
res0: AccountStatus = Active
This reader isn't perfect though, mainly because it's not handling the error your code will give you on out of bound numbers:
scala> Json.parse("""{"as":20}""").as[AccountStatus]
java.lang.RuntimeException: Out of bound AccountStatus Value.
at scala.sys.package$.error(package.scala:27)
at AccountStatus$.fromInt(<console>:42)
at AccountStatusReads$$anonfun$reads$1.apply(<console>:27)
at AccountStatusReads$$anonfun$reads$1.apply(<console>:27)
at play.api.libs.json.JsResult$class.map(JsResult.scala:81)
at play.api.libs.json.JsSuccess.map(JsResult.scala:9)
at AccountStatusReads$.reads(<console>:27)
at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
at play.api.libs.json.JsObject.as(JsValue.scala:166)
... 42 elided
You could handle this by making the Reads handle the error. I can show you how if you want, but first the other part of a Format is a Writes. This trait, unsurprisingly is similar to reads except it does the reverse. You're taking your class AccountStatus and creating a JsValue (JSON). So, you just have to implement the writes method.
implicit object AccountStatusWrites extends Writes[AccountStatus] {
def writes(as: AccountStatus): JsValue = {
JsObject(Seq("as" -> JsNumber(as.as)))
}
}
Then this can be used to serialize that class to JSON like so:
scala> Json.toJson(Draft)
res4: play.api.libs.json.JsValue = {"as":0}
Now, this is actually enough to get your error to go away. Why? Because Json.format[Account] is doing all the work we just did for you! But for Account. It can do this because it's a case class and has less than 22 fields. Also every field for Account has a way to be converted to and from JSON (via a Reads and Writes). Your error message was showing that Account could not have a format automatically created for it because part of it (status field) had no formatter.
Now, why do you have to do this? Because AccountStatus is not a case class, so you can't call Json.format[AccountStatus] on it. And because the subclasses of it are each objects, which have no unapply method defined for them since they're not case classes. So you have to explain to the library how to serialize and deserialize.
Since you said you're new to scala, I imagine that the concept of an implicit is still somewhat foreign. I recommend you play around with it / do some reading to get a grasp of what to do when you see that the compiler is complaining about not being able to find an implicit it needs.
Bonus round
So, you might really not want to do that work yourself, and there is a way to avoid having to do it so you can do Json.format[AccountStatus]. You see Json.format uses the apply and unapply methods to do its dirty work. In scala, these two methods are defined automatically for case classes. But there's no reason you can't define them yourself and get everything they give you for free!
So, what do apply and unapply look like type signature wise? It changes per class, but in this case apply should match Int => AccountStatus (a function that goes from an int to an AccountStatus). So it's defined like so:
def apply(i: Int): AccountStatus = fromInt(i)
and unapply is similar to the reverse of this, but it needs to return an Option[Int], so it looks like
def unapply(as: AccountStatus): Option[Int] = Option(as.as)
with both of these defined you don't need to define the reads and writes yourself and instead can just call
// this is still inside the AccountStatus object { ... }
implicit val asFormat = Json.format[AccountStatus]
and it will work in a similar fashion.
.P.S. I'm traveling today, but feel free to leave any comments if some of this doesn't make sense and I'll try to get back to you later on
I apologize in advance if this is an XY problem.
tl;dr:
I'd like to have a compile-time map of type [Request.type, Response.type] so I can effectively say if I send message Request, a CLI should, at compile-time, know how to deserialize its expected Response, irrespective of the fact that it won't know what type of request is sent until runtime.
too long; still read:
I have a CLI which communicates with an HTTP server and depending on the type of message sent to the HTTP server, I'd like to validate the JSON response against a case case.
For instance, if I send the HTTP server an AddFoo message, I might want to validate that the JSON response can be deserialized into an AddedFoo, etc.
My current solution is quite hacky. Using play-json, I'm attempting to parse the JSON response using a mapping from config.mode (i.e., command issued to the CLI) to the expected responses' implicit Reads.
My code looks something like this:
val modeToResponseReads: Map[String, Reads[_]] = Map(
Modes.ADD_FOO -> AddedFoo.addedFooReads,
Modes.ADD_BOO -> AddedBoo.addedBooReads,
Modes.GET_WOO -> GetWooResponse.getWooReads,
)
parser.parse(args, MyConfig()) match {
case Some(config) => try {
val exec = new MyHttpExecutor(remoteUri, config)
val res = Await.result(exec.getResponse, 100.seconds)
// passing `Reads` to `as` because JsValue#as[T] cannot be
// applied at runtime -- only compile-time.
val _ = Json.parse(res.json.toString)
.as(modeToResponseReads(config.mode))
exec.actorSystem.terminate()
exec.wsClient.close()
} catch {
case t: Throwable => logger.error(t.getMessage)
}
case None => {
logger.error("Bad arguments.")
sys.exit(1)
}
}
While this works, it's an incredible kludge that becomes increasingly unmaintainable with an increasing number of messages. Further, I've found that this pattern will need to be replicated anywhere some type of validate or conversion will need to happen (e.g., Future[Any] being converted to Future[AddedFoo]).
Surely my approach isn't the right way... how is this traditionally done? If it is the right way (please no), are there optimizations that can be made?
I managed to accomplish this by encoding the contract directly into the child Request classes. Namely, the child Request classes would hold a ResponseType type with the base class enforcing the covariant type.
So I can do something like this:
abstract class Response
abstract class Request[+A <: Response]
case class Foo(id: String)
object Foo {
implicit val fooReads = Json.reads[Foo]
implicit val fooFormat = Json.format[Foo]
}
case class FooResponse(foo: Foo) extends Response {
def greet = println("woo hoo!")
}
object FooResponse {
implicit val fooRespReads = Json.reads[FooResponse]
implicit val fooRespFormat = Json.format[FooResponse]
}
case class FooRequest() extends Request[FooResponse] {
type ResponseType = FooResponse
}
object Main extends App {
val req: FooRequest = new FooRequest()
val foo = Foo("12345")
val resp = new FooResponse(foo)
val respJsonString = Json.toJson(resp).toString
println(Json.parse(respJsonString).as[req.ResponseType])
}
I have some simple messages with implicit Json.reads and Json.formats defined in their companion objects. All of these messages extend MyBaseMessage.
In other words, for any T <: MyBaseMessage, T is (de)serializable.
These messages represent simple CRUD operations to be performed on a cluster, so there's an Play server that will sit between a CLI sending JSON and the Cluster. Because the operations are simple, I should be able to make some very generic Actions on the Play side: when I receive JSON at an endpoint, deserialize the message according to the endpoint and forward that message to the cluster.
My ultimate goal is to do something like this:
// AddBooMessage extends MyBaseMessage
def addBoo = FooAction[AddBooMessage]
// AddMooMessage extends MyBaseMessage
def addMoo = FooAction[AddMooMessage]
// etc. ...
So when a request is sent to the route corresponding to the addBoo message, the request's JSON will be parsed into an AddBooMessage message and pushed to the cluster. Repeat ad nauseam.
I have the following written:
private def FooAction[T <: MyBaseMessage] = Action {
implicit request =>
parseAndForward[T](request)
}
private def parseAndForward[T <: MyBaseMessage](request: Request[AnyContent]) = {
val parsedRequest = Json.parse(request.body.toString).as[T]
Logger.info(s"Got '$parsedRequest' request. Forwarding it to the Cluster.")
sendToCluster(parsedRequest)
}
But I find the following error:
No Json deserializer found for type T. Try to implement an implicit Reads or Format for this type.
However, all of these messages are serializable and have both Reads and Format defined for them.
I tried passing (implicit fjs: Reads[T]) to parseAndForward in hopes to implicitly provide the Reads required (though it should already be implicitly provided), but it didn't help.
How can I solve this problem?
JsValue#as[A] needs an implicit Reads[A] in order to deserialize JSON to some type A. That is, the error message you're getting is caused because the compiler can't guarantee there is a Reads[T] for any type T <: MyBaseMessage. Assuming sendToCluster is parameterized the same way, this can easily by fixed by simply requiring an implicit Reads[T] in each method call. It sounds like you were close, and just needed to take things a step further by requiring the Reads[T] from FooAction, as well (since that call is where the type is determined).
private def FooAction[T <: MyBaseMessage : Reads] = Action { implicit request =>
parseAndForward[T](request)
}
private def parseAndForward[T <: MyBaseMessage : Reads](request: Request[AnyContent]) = {
val parsedRequest = Json.parse(request.body.toString).as[T]
Logger.info(s"Got '$parsedRequest' request. Forwarding it to the Cluster.")
sendToCluster(parsedRequest) // Assuming this returns a `Future[Result]`
}
If your intention if you use the above code by manually supplying the type parameter, this will work just fine.
There are some other improvements I think you can make here. First, if you always expect JSON, you should require the parse.json BodyParser. This will return a BadRequest if what is received isn't even JSON. Second, as will throw an exception if the received JSON cannot be deserialized into the expected type, you can use JsValue#validate to do this more safely and fold the result to handle the success and error cases explicitly. For example:
private def FooAction[T <: MyBaseMessage] = Action.async(parse.json) { implicit request =>
parseAndForward[T](request)
}
private def parseAndForward[T <: MyBaseMessage](request: Request[JsValue]) = {
request.body.validate[T].fold(
error => {
Logger.error(s"Error parsing request: $request")
Future.successful(BadRequest)
},
parsed => {
Logger.info(s"Got '$parsed' request. Forwarding it to the Cluster.")
sendToCluster(parsed)
}
)
}
I've recently seen code like this:
val maybeInt = catching(classOf[NFE]) opt arg.toInt
What is this opt? An Option? Why isn't it using getOrElse to extract the value? In the above code, will maybeInt be None if a NumberFormatException gets thrown?
catching looks like it's some sort of method call, doesn't it? It is, but it actually returns an instance of a class Catch; it doesn't directly take an argument. This class has two methods that are particularly useful for dealing with exceptions (and several more for catching multiple exceptions). The first is
def opt [U >: T] (body: ⇒ U) : Option[U]
which is being used here--you give it something that may throw an exception, and it will return Some(result) if everything went okay, and None if the targeted exception was caught:
scala> type NFE = NumberFormatException
defined type alias NFE
scala> import scala.util.control.Exception._
import scala.util.control.Exception._
scala> catching(classOf[NFE]).opt( "fish".toInt )
res0: Option[Int] = None
scala> catching(classOf[NFE]).opt( "42".toInt )
res1: Option[Int] = Some(42)
You can then deal with this with map or filter or getOrElse or whatever else you use to deal with options.
The other useful method is either, which returns an instance of Left(exception) if an exception was thrown, and a Right(result) if it was not:
scala> catching(classOf[NFE]).either( "fish".toInt )
res2: Either[Throwable,Int] = Left(java.lang.NumberFormatException: For input string: "fish")
scala> catching(classOf[NFE]).either( "42".toInt )
res3: Either[Throwable,Int] = Right(42)
You can then use fold or map to an option or whatever else you like doing with eithers.
Note that you can define a single catcher and use it multiple times (so you don't need to create the catcher object every time you, for example, parse an integer):
scala> val catcher = catching(classOf[NFE])
catcher: util.control.Exception.Catch[Nothing] = Catch(java.lang.NumberFormatException)
scala> catcher.opt("42".toInt)
res4: Option[Int] = Some(42)
scala> catcher.opt("fish".toInt)
res5: Option[Int] = None
Edit: as Daniel points out in the comments, this still creates a temporary Catch[Option]; given the method signatures, there isn't an easy way to just have it trap exceptions and generate options without creating any extra objects. This reminds me why I write my own methods to do exactly that:
def optNFE[T](t: => T) = try { Some(t) } catch {case nfe: NFE => None}
optNFE( "fish".toInt ) // gives None
optNFE( "42".toInt ) // gives Some(42)
I use a more simple pattern when there is only one catch :
try{
return args.split(" ").exists(line.startsWith _)
}catch {
case _ =>{//generic exception
logger.error("Error with line ${line} for ${ex.message}")
throw _
}
}
I'm definitely not yet a Scala pro, and I guess you could find shorter stuff