Extending AutoDerivation in Circe does not work - json

My question concerns the second solution offered by mixel here: Scala Circe with generics
Note that the trait named Auto in Circe has been renamed to AutoDerivation in the current version of Circe.
I am using the solution mixel provides in his StackOverflow solution but have not been able to get it to work. I have tried things like updating my Circe version to the most recent one and making sure the Macro Paradise plugin is imported, but still no luck.
Here is my code. The first is its own file, called CirceGeneric.
import io.circe._
import io.circe.parser._
import io.circe.generic.extras._
object CirceGeneric {
trait JsonEncoder[T] {
def apply(in: T): Json
}
trait JsonDecoder[T] {
def apply(s: String): Either[Error, T]
}
object CirceEncoderProvider {
def apply[T: Encoder]: JsonEncoder[T] = new JsonEncoder[T] {
def apply(in: T) = Encoder[T].apply(in)
}
}
object CirceDecoderProvider {
def apply[T: Decoder]: JsonDecoder[T] = new JsonDecoder[T] {
def apply(s: String) = decode[T](s)
}
}
}
object Generic extends AutoDerivation {
import CirceGeneric._
implicit def encoder[T: Encoder]: JsonEncoder[T] = CirceEncoderProvider[T]
implicit def decoder[T: Decoder]: JsonDecoder[T] = CirceDecoderProvider[T]
}
The second is a method for unit testing that uses the Akka function responseAs. The method appears in a class called BaseServiceTest.
def responseTo[T]: T = {
def response(s: String)(implicit d: JsonDecoder[T]) = {
d.apply(responseAs[String]) match {
case Right(value) => value
case Left(error) => throw new IllegalArgumentException(error.fillInStackTrace)
}
}
response(responseAs[String])
}
The idea is to convert the result of responseAs[String] (which returns a string) into a decoded case class.
The code is not behaving as expected. Intellij does not detect any missing implicits, but when compilation time comes around, I am getting problems. I should mention that the BaseServiceTest file contains imports for CirceGeneric._ and Generic._, so a missing import statement is not the problem.
[error] [...]/BaseServiceTest.scala:59: could not find implicit value for parameter d: [...]CirceGeneric.JsonDecoder[T]
[error] response(responseAs[String])
Either the implicit conversion from Decoder[T] to JsonDecoder[T] is not happening, or the Decoder[T] instance is not being created. Either way, something is wrong.

You still need a Decoder or JsonDecoder context bound on responseTo.
def responseTo[T : Decoder]: T = ...
This is because all your code, and indeed mixel's code in the linked answer, is about abstracting from a Decoder out to a JsonDecoder trait which can be used for cross-library support. But you still don't have any way of constructing one without an underlying Decoder instance.
Now, there are some ways of automatically generating Decoders for (for instance) case classes contained in circe.generics.auto, but at this point in your code
def responseTo[T]: T = {
def response(s: String)(implicit d: JsonDecoder[T]) = ...
...
}
you're asking the compiler to be able to provide an implicit JsonDecoder (i.e., in your setup, Decoder) instance for any arbitrary type. As the accepted answer to the linked question explains, that's not possible.
You need to delay the implicit resolution to the point where you know what type you're dealing with - in particular, that you can provide a Decoder[T] instance for it.
EDIT: In your response to your comment regarding what the point is if you can't create JsonDecoders for all types...
My understanding of the linked question is that they're trying to abstract away the circe library in order to allow swapping out the JSON library implementation. This is done as follows:
add the JsonDecoder type class
have a package json which contains implicits (using Circe) for constructing them automatically via the package object extending AutoDerivation
have external code only refer to JsonDecoder and import the implicits in the json package
Then all the JSON serialization and implicit resolution works out without ever needing the calling code to reference io.circe, and it's easy to switch over the json/JsonDecoder to another JSON library if desired. But you're still going to have to use the JsonDecoder context bound, and be restricted to working with types where such an implicit can be constructed. Which is not every type.

Related

How to add a custom marshaller to akka http?

As a beginner to both scala and akka-http, I am trying to hook into the serialization aka marshalling process.
The project uses akka#2.5.2 and akka-http#10.0.10". Furthermore, it has the akka-http-spray-json dependency included.
In the codebase, we use Java.Util.Currency (It may be deprecated which is not important as I'd still like to know how to add a custom marshaller.)
Given this example controller:
def getCurrencyExample: Route = {
path("currencyExample") {
val currency: Currency = Currency.getInstance("EUR")
val code: String = currency.getCurrencyCode
val shouldBeFormated = collection.immutable.HashMap(
"currencyCode" -> code,
"currencyObject" -> currency
)
complete(shouldBeFormated)
}
}
I get a response like this where the currency object becomes empty:
{
currencyObject: { },
currencyCode: "EUR",
}
I expect something like:
{
currencyObject: "EUR",
currencyCode: "EUR",
}
The currency object should be transformed into a JSON string. And since I do not want to transform each response manually, I want to hook into marshalling process and have it done in the background.
I want to add a custom marhaller only for Java.Util.Currency objects, yet even reading up on the docs I am very unsure how to proceed.
There are multiple approaches described, and I am not sure which fits my need, or where to start.
I tried creating my own CurrencyJsonProtocol:
package com.foo.api.marshallers
import java.util.Currency
import spray.json.{DefaultJsonProtocol, JsString, JsValue, RootJsonFormat}
object CurrencyJsonProtocol extends DefaultJsonProtocol {
implicit object CurrencyJsonFormat extends RootJsonFormat[Currency] {
override def read(json: JsValue): Currency = {
Currency.getInstance(json.toString)
}
override def write(obj: Currency): JsValue = {
JsString(obj.getCurrencyCode)
}
}
}
yet the mere existence of the file breaks my project:
[error] RouteDefinitons.scala:90:16: type mismatch;
[error] found : scala.collection.immutable.HashMap[String,java.io.Serializable]
[error] required: akka.http.scaladsl.marshalling.ToResponseMarshallable
[error] complete(shouldBeFormated)
[error] ^
[error] one error found
[error] (Compile / compileIncremental) Compilation failed
and I have no idea why. (It was crashing due to my package name being called marshaller. That completely broke the compilation of the project.
From what I understand, you have all the pieces, you just need to put them together.
Spray json provides support for marshalling common types, such as Int, String, Boolean, List, Map, etc. But it doesn't know how to marshall 'Currency'. And to solve that you have created a custom marshaller for 'Currency' objects. You just need to plug it in the right place. All that you have to do is import the marshaller from your CurrencyJsonProtocol into your Controller like below:
import CurrencyJsonProtocol._
Also make sure you have the below imports as well:
import spray.httpx.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._
And spray-json should automatically pick that up.
To understand how it works, you need to understand about implicits in scala. Although it looks like magic when you come from java-world like me, I can assure you it's not.
I´m using Spray support for years. Maybe i should try another one. Anyway, in Spray, a custom formatter for a type is simple. An example of the squants.market.currency https://github.com/typelevel/squants
import squants.market.{ Currency, Money }
import squants.market._
trait MoneyJsonProtocol extends DefaultJsonProtocol {
implicit object CurJsonFormat extends JsonFormat[Currency] {
def write(currency: Currency): JsValue = JsString(currency.code)
def read(value: JsValue): Currency = value match {
case JsString(currency) =>
defaultCurrencyMap.get(currency) match {
case Some(c) => c
case None => // throw an exception, for example
}
case _ => // throw another exception, for example
}
}
implicit def toJavaBigDecimal(b: math.BigDecimal): java.math.BigDecimal = b.underlying
}
Do you have a JsonFormat[Currency]? If yes, just fix it...
If not, you should follow the doc and create:
as domain model, specific result type to hold the answer e.g. case class CurrencyResult(currency: Currency, code: String)
in trait JsonSupport: a JsonFormat[Currency]
in trait JsonSupport: a JsonFormat[CurrencyResult]
As an alternative to point 1 above, you can probably type your map as: Map[String, JsValue].

No instance of play.api.libs.json.Format is available for models.AccountStatus in the implicit scope

No instance of play.api.libs.json.Format is available for models.AccountStatus in the implicit scope.
This is the code taken from a github page, and only class names and variable names are changed.
package models
import slick.jdbc.H2Profile._
import play.api.libs.json._
case class Account(id: Long, name: String, category: Int, status:AccountStatus)
object Account {
implicit val accountFormat = Json.format[Account]
}
sealed abstract class AccountStatus(val as:Int)
object AccountStatus{
final case object Draft extends AccountStatus(0)
final case object Active extends AccountStatus(1)
final case object Blocked extends AccountStatus(2)
final case object Defaulter extends AccountStatus(3)
implicit val columnType: BaseColumnType[AccountStatus] = MappedColumnType.base[AccountStatus,Int](AccountStatus.toInt, AccountStatus.fromInt)
private def toInt(as:AccountStatus):Int = as match {
case Draft => 0
case Active => 1
case Blocked => 2
case Defaulter => 3
}
private def fromInt(as: Int): AccountStatus = as match {
case 0 => Draft
case 1 => Active
case 2 => Blocked
case 3 => Defaulter
_ => sys.error("Out of bound AccountStatus Value.")
}
}
https://github.com/playframework/play-scala-slick-example/blob/2.6.x/app/models/Person.scala
So, this code needs to be added inside of the object AccountStatus code block since we need to use fromInt to transform an Int to an AccountStatus. This is a Reads defined for AccountStatus:
implicit object AccountStatusReads extends Reads[AccountStatus] {
def reads(jsValue: JsValue): JsResult[AccountStatus] = {
(jsValue \ "as").validate[Int].map(fromInt)
}
}
What's a Reads? It's just a trait that defines how a JsValue (the play class encapsulating JSON values) should be deserialized from JSON to some type. The trait only requires one method to be implemented, a reads method which takes in some json and returns a JsResult of some type. So you can see in the above code that we have a Reads that will look for a field in JSON called as and try to read it as an integer. From there, it will then transform it into an AccountStatus using the already defined fromInt method. So for example in the scala console you could do this:
import play.api.libs.json._
// import wherever account status is and the above reader
scala> Json.parse("""{"as":1}""").as[AccountStatus]
res0: AccountStatus = Active
This reader isn't perfect though, mainly because it's not handling the error your code will give you on out of bound numbers:
scala> Json.parse("""{"as":20}""").as[AccountStatus]
java.lang.RuntimeException: Out of bound AccountStatus Value.
at scala.sys.package$.error(package.scala:27)
at AccountStatus$.fromInt(<console>:42)
at AccountStatusReads$$anonfun$reads$1.apply(<console>:27)
at AccountStatusReads$$anonfun$reads$1.apply(<console>:27)
at play.api.libs.json.JsResult$class.map(JsResult.scala:81)
at play.api.libs.json.JsSuccess.map(JsResult.scala:9)
at AccountStatusReads$.reads(<console>:27)
at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
at play.api.libs.json.JsObject.as(JsValue.scala:166)
... 42 elided
You could handle this by making the Reads handle the error. I can show you how if you want, but first the other part of a Format is a Writes. This trait, unsurprisingly is similar to reads except it does the reverse. You're taking your class AccountStatus and creating a JsValue (JSON). So, you just have to implement the writes method.
implicit object AccountStatusWrites extends Writes[AccountStatus] {
def writes(as: AccountStatus): JsValue = {
JsObject(Seq("as" -> JsNumber(as.as)))
}
}
Then this can be used to serialize that class to JSON like so:
scala> Json.toJson(Draft)
res4: play.api.libs.json.JsValue = {"as":0}
Now, this is actually enough to get your error to go away. Why? Because Json.format[Account] is doing all the work we just did for you! But for Account. It can do this because it's a case class and has less than 22 fields. Also every field for Account has a way to be converted to and from JSON (via a Reads and Writes). Your error message was showing that Account could not have a format automatically created for it because part of it (status field) had no formatter.
Now, why do you have to do this? Because AccountStatus is not a case class, so you can't call Json.format[AccountStatus] on it. And because the subclasses of it are each objects, which have no unapply method defined for them since they're not case classes. So you have to explain to the library how to serialize and deserialize.
Since you said you're new to scala, I imagine that the concept of an implicit is still somewhat foreign. I recommend you play around with it / do some reading to get a grasp of what to do when you see that the compiler is complaining about not being able to find an implicit it needs.
Bonus round
So, you might really not want to do that work yourself, and there is a way to avoid having to do it so you can do Json.format[AccountStatus]. You see Json.format uses the apply and unapply methods to do its dirty work. In scala, these two methods are defined automatically for case classes. But there's no reason you can't define them yourself and get everything they give you for free!
So, what do apply and unapply look like type signature wise? It changes per class, but in this case apply should match Int => AccountStatus (a function that goes from an int to an AccountStatus). So it's defined like so:
def apply(i: Int): AccountStatus = fromInt(i)
and unapply is similar to the reverse of this, but it needs to return an Option[Int], so it looks like
def unapply(as: AccountStatus): Option[Int] = Option(as.as)
with both of these defined you don't need to define the reads and writes yourself and instead can just call
// this is still inside the AccountStatus object { ... }
implicit val asFormat = Json.format[AccountStatus]
and it will work in a similar fashion.
.P.S. I'm traveling today, but feel free to leave any comments if some of this doesn't make sense and I'll try to get back to you later on

Scala Circe with generics

I am trying to use the scala json library Circe, wrapping it in a simple trait to provide conversion to/from json for which I have the following:
import io.circe.generic.auto._
import io.circe.parser._
import io.circe.syntax._
trait JsonConverter {
def toJson[T](t : T) : String
def fromJson[T](s: String) : T
}
case class CirceJsonConverter() extends JsonConverter{
override def toJson[T](t: T): String = t.asJson.noSpaces
override def fromJson[T](s: String): T = decode[T](s).getOrElse(null).asInstanceOf[T]
}
The aim of this is to simply be able to call JsonConverter with any object and have it convert it to/from json as such jsonConverter.toJson(0) must equalTo("0") , however when I try to compile it I get the following:
[error] could not find implicit value for parameter encoder: io.circe.Encoder[T]
[error] override def toJson[T](t: T): String = t.asJson.noSpaces
[error] ^
[error] could not find implicit value for parameter decoder: io.circe.Decoder[T]
[error] override def fromJson[T](s: String): T = decode[T](s).getOrElse(null).asInstanceOf[T]
[error] ^
[error] two errors found
I can of course have a class that everything I intend to put through the converter inherit from, but I had the impression that circe could auto generate the encoders/decoders?
What you want is not going to work unless you can implement a strategy for turning any object into Json... which seems unlikely. Circe (and many other libs) instead choose to use a common pattern called Type Classes to make it convenient to define how you want to do something, in this case Encoder/Decoder, for a specific type.
I recommend researching Type Classes if you are unfamiliar with them. And then take a look at the Circe docs to see how you can implement Encoders/Decoders specifically.
Following Idan Waisman answer and C4stor answer in my duplicate question I used the Type Classes pattern. For brevity I provide sample code only for decoding json. Encoding can be implemented in exactly the same way.
First, let's define the trait that will be used to inject json decoder dependency:
trait JsonDecoder[T] {
def apply(s: String): Option[T]
}
Next we define object that creates instance implementing this trait:
import io.circe.Decoder
import io.circe.parser.decode
object CirceDecoderProvider {
def apply[T: Decoder]: JsonDecoder[T] =
new JsonDecoder[T] {
def apply(s: String) =
decode[T](s).fold(_ => None, s => Some(s))
}
}
As you can notice apply requires implicit io.circe.Decoder[T] to be in scope when it called.
Then we copy io.circe.generic.auto object content and create a trait (I made PR to have this trait available as io.circe.generic.Auto):
import io.circe.export.Exported
import io.circe.generic.decoding.DerivedDecoder
import io.circe.generic.encoding.DerivedObjectEncoder
import io.circe.{ Decoder, ObjectEncoder }
import io.circe.generic.util.macros.ExportMacros
import scala.language.experimental.macros
trait Auto {
implicit def exportDecoder[A]: Exported[Decoder[A]] = macro ExportMacros.exportDecoder[DerivedDecoder, A]
implicit def exportEncoder[A]: Exported[ObjectEncoder[A]] = macro ExportMacros.exportEncoder[DerivedObjectEncoder, A]
}
Next in the package (e.g. com.example.app.json) that uses json decoding a lot we create package object if does not exist and make it extend Auto trait and provide implicit returning JsonDecoder[T] for given type T:
package com.example.app
import io.circe.Decoder
package object json extends Auto {
implicit def decoder[T: Decoder]: JsonDecoder[T] = CirceDecoderProvider[T]
}
Now:
all source files in com.example.app.json has Auto implicits in scope
you can get JsonDecoder[T] for any type T that has io.circe.Decoder[T] or for which it can be generated with Auto implicits
you do not need to import io.circe.generic.auto._ in every file
you can switch between json libraries by only changing com.example.app.json package object content.
For example you can switch to json4s (though I did the opposite and switched to circe from json4s). Implement provider for JsonDecoder[T]:
import org.json4s.Formats
import org.json4s.native.JsonMethods._
import scala.util.Try
case class Json4SDecoderProvider(formats: Formats) {
def apply[T: Manifest]: JsonDecoder[T] =
new JsonDecoder[T] {
def apply(s: String) = {
implicit val f = formats
Try(parse(s).extract[T]).toOption
}
}
}
And change com.example.app.json package object content to:
package com.example.app
import org.json4s.DefaultFormats
package object json {
implicit def decoder[T: Manifest]: JsonDecoder[T] = Json4SDecoderProvider(DefaultFormats)[T]
}
With Type Classes pattern you get compile-time dependency injection. That gives you less flexibility than runtime dependency injection but I doubt that you need to switch json parsers in runtime.

Explicitly output JSON null in case of missing optional value

Consider this example using Play's JSON API (play.api.libs.json):
case class FooJson(
// lots of other fields omitted
location: Option[LocationJson]
)
object FooJson {
implicit val writes = Json.writes[FooJson]
}
and
case class LocationJson(latitude: Double, longitude: Double)
object LocationJson {
implicit val writes = Json.writes[LocationJson]
}
If location is None, the resulting JSON won't have location field at all. This is fine and understadable. But if I wanted for some reason (say, to make my API more self-documenting), how can I explicitly output null in JSON?
{
"location": null
}
I also tried defining the field as location: LocationJson and passing option.orNull to it, but it does not work (scala.MatchError: null at play.api.libs.json.OWrites$$anon$2.writes). For non-custom types such as String or Double, this approach would produce null in JSON output.
So, while using Json.writes[FooJson] as shown above (or something equally simple, i.e. not having to write a custom Writes implementation), is there a clean way to include nulls in JSON?
What I'm asking is analogous to JsonInclude.Include.ALWAYS in the Jackson library (also Jackson's default behaviour). Similarly in Gson this would be trivial
(new GsonBuilder().serializeNulls().create()).
Play 2.4.4
Greg Methvin, a Play committer, wrote this answer to me in a related GitHub issue:
The JSON macros only support one way of encoding optional values,
which is to omit None values from the JSON. This is not a bug but
rather a limitation of the implementation. If you want to include
nulls you're unfortunately going to have to implement your own Writes.
I do think we should try to provide more configurability for the
macros though.
In this case, I'll let Play exclude this field when the value is null, even if it slightly sacrifices API consistency and self-documentability. It's still such a minor thing (in this particular API) that it doesn't warrant uglifying the code as much as a custom Writes would take for a case class with a dozen values.
I'm hoping they do make this more configurable in future Play versions.
Hello from the future.
As of Play 2.7, a fairly simple solution was introduced for automated JSON codecs. We can introduce the appropriate implicit value for JsonConfiguration in the scope for the Format/Reads/Writes. The following configuration will write nulls for empty Options instead of omitting the fields entirely.
import play.api.libs.json._
implicit val config = JsonConfiguration(optionHandlers = OptionHandlers.WritesNull)
implicit val residentWrites = Json.writes[Resident]
Reference
Here's a way to do it:
object MyWrites extends DefaultWrites{
override def OptionWrites[T](implicit fmt: Writes[T]): Writes[Option[T]] = new Writes[Option[T]] {
override def writes(o: Option[T]): JsValue = {
o match {
case Some(a) => Json.toJson(a)(fmt)
case None => JsNull
}
}
}
}
This will overwrite the default implementation which will not create an element. I used this in your sample code:
case class FooJson(
// ...
location: Option[LocationJson]
)
case class LocationJson(latitude: Double, longitude: Double)
object LocationJson {
implicit val writes = Json.writes[LocationJson]
}
implicit val fooJsonWriter: Writes[FooJson] = new Writes[FooJson] {
override def writes(o: FooJson): JsValue = {
JsObject(Seq(
"location" -> Json.toJson(o.location)
// Additional fields go here.
))
}
}
Json.toJson(FooJson(None))
And got this result res0: play.api.libs.json.JsValue = {"location":null}.
if we have null values then we have to add the option with members in case class which will resolve the issue
case class response(
name:String,
age: option[int]
)
object response {
implicit val format = Json.format[response]
}
Here the option is the answer for us. and if we are the JSON response for age is coming as null and this will handle the solution for us.

Error working with JSON in Play 2

I am trying to serialize / deserialize using JSON and the Play Framework 2.1.0 and Scala 2.10. I am using Anorm, and I have a pretty simple Object that I want to store in a database. The Order is very simple:
case class Order(id: Pk[Long] = NotAssigned, mfg: String, tp: String)
In my controller, I am trying to build a REST interface to be able to accept and send an Order instance (above) as JSON. In there, I have the following code:
implicit object PkFormat extends Format[Pk[Long]] {
def reads(json: JsValue):Pk[Long] = Id(json.as[Long])
def writes(id: Pk[Long]):JsNumber = JsNumber(id.get)
}
However, this is failing to compile when I run "play test" with the following:
overriding method reads in trait Reads of type (json: play.api.libs.json.JsValue)play.api.libs.json.JsResult[anorm.Pk[Long]];
[error] method reads has incompatible type
[error] def reads(json: JsValue):Pk[Long] = Id(json.as[Long])
Does anyone know why this is happening?
I have a lot of experience with JAXB, but I am realtively new to Play and Scala, and I haven't been able to find any answers so far. This seems like a pretty simple use case, actually I was hoping there would be a simpler solution (like Annotations), but I wasn't able to find one (at least not yet)
Any help is greatly appreciated!
Thanks
play.api.libs.json.Reads trait defines reads method as:
def reads(json : play.api.libs.json.JsValue) : play.api.libs.json.JsResult[A]
Therefore, the response of reads method is expected to be JsResult[A], not A; that is, JsResult[Pk[Long]] and not Pk[Long]. In a case of success, you'll want to return this:
implicit object PkFormat extends Format[Pk[Long]] {
def reads(json: JsValue):JsResult[Pk[Long]] = JsSuccess(Id(json.as[Long]))
def writes(id: Pk[Long]):JsNumber = JsNumber(id.get)
}