I am using an external service and I am expecting to receive a json containing a field with three kinds of value:
A Double
null
The field could be absent
I need to deserialize the json into a case class and somewhere else in the code I need to serialize it to a json with the same fields.
I have an implicit reads and write:
implicit lazy val aReads: Reads[A] = (
"foo".readNullable[Double]
)(A.apply _)
implicit lazy aWrites: OWrites[A] = (
"foo".write[Option[Double]]
)
and the case class:
case class A(
foo: Option[Double]
)
As you can imagine the problem is that I am not able to "catch" when the value is absent and if a use "foo".writeNullable[Double] as writers I am not able to catch when it's null (it will be always absent). How can I solve this issue?
What you need is actually a data type which reflects three states:
Present
Null
Non Existing
There is actually a nice encoding of this here, which is semantically equivalent to:
sealed trait Tristate[+A]
case class Present[+A](a: A) extends Tristate[A]
case object Absent extends Tristate[Nothing] // this can represent your "null" state
case object NonExisting extends Tristate[Nothing]
The small library linked has nice combinators over Tristate such as map, flatMap, filter, etc.
And then, you can derive a Play decoder/encoder which puts the object in the correct state serializes appropriately.
Related
I'm trying to construct a trait and an abstract class to subtype by messages (In an Akka play environment) so I can easily convert them to Json.
What have done so far:
abstract class OutputMessage(val companion: OutputMessageCompanion[OutputMessage]) {
def toJson: JsValue = Json.toJson(this)(companion.fmt)
}
trait OutputMessageCompanion[OT] {
implicit val fmt: OFormat[OT]
}
Problem is, when I'm trying to implement the mentioned interfaces as follows:
case class NotifyTableChange(tableStatus: BizTable) extends OutputMessage(NotifyTableChange)
object NotifyTableChange extends OutputMessageCompanion[NotifyTableChange] {
override implicit val fmt: OFormat[NotifyTableChange] = Json.format[NotifyTableChange]
}
I get this error from Intellij:
Type mismatch, expected: OutputMessageCompanion[OutputMessage], actual: NotifyTableChange.type
I'm kinda new to Scala generics - so help with some explanations would be much appreciated.
P.S I'm open for any more generic solutions than the one mentioned.
The goal is, when getting any subtype of OutputMessage - to easily convert it to Json.
The compiler says that your companion is defined over the OutputMessage as the generic parameter rather than some specific subtype. To work this around you want to use a trick known as F-bound generic. Also I don't like the idea of storing that companion object as a val in each message (after all you don't want it serialized, do you?). Defining it as a def is IMHO much better trade-off. The code would go like this (companions stays the same):
abstract class OutputMessage[M <: OutputMessage[M]]() {
self: M => // required to match Json.toJson signature
protected def companion: OutputMessageCompanion[M]
def toJson: JsValue = Json.toJson(this)(companion.fmt)
}
case class NotifyTableChange(tableStatus: BizTable) extends OutputMessage[NotifyTableChange] {
override protected def companion: OutputMessageCompanion[NotifyTableChange] = NotifyTableChange
}
You may also see standard Scala collections for an implementation of the same approach.
But if all you need the companion for is to encode with JSON format, you can get rid of it like this:
abstract class OutputMessage[M <: OutputMessage[M]]() {
self: M => // required to match Json.toJson signature
implicit protected def fmt: OFormat[M]
def toJson: JsValue = Json.toJson(this)
}
case class NotifyTableChange(tableStatus: BizTable) extends OutputMessage[NotifyTableChange] {
override implicit protected def fmt: OFormat[NotifyTableChange] = Json.format[NotifyTableChange]
}
Obviously is you also want to decode from JSON you still need a companion object anyway.
Answers to the comments
Referring the companion through a def - means that is a "method", thus defined once for all the instances of the subtype (and doesn't gets serialized)?
Everything you declare with val gets a field stored in the object (instance of the class). By default serializers trying to serialize all the fields. Usually there is some way to say that some fields should be ignored (like some #IgnoreAnnotation). Also it means that you'll have one more pointer/reference in each object which uses memory for no good reason, this might or might not be an issue for you. Declaring it as def gets a method so you can have just one object stored in some "static" place like companion object or build it on demand every time.
I'm kinda new to Scala, and I've peeked up the habit to put the format inside the companion object, would you recommend/refer to some source, about how to decide where is best to put your methods?
Scala is an unusual language and there is no direct mapping the covers all the use cases of the object concept in other languages. As a first rule of thumb there are two main usages for object:
Something where you would use static in other languages, i.e. a container for static methods, constants and static variables (although variables are discouraged, especially static in Scala)
Implementation of the singleton pattern.
By f-bound generic - do you mean the lower bound of the M being OutputMessage[M] (btw why is it ok using M twice in the same expr. ?)
Unfortunately wiki provides only a basic description. The whole idea of the F-bounded polymorphism is to be able to access to the type of the sub-class in the type of a base class in some generic manner. Usually A <: B constraint means that A should be a subtype of B. Here with M <: OutputMessage[M], it means that M should be a sub-type of the OutputMessage[M] which can easily be satisfied only by declaring the child class (there are other non-easy ways to satisfy that) as:
class Child extends OutputMessage[Child}
Such trick allows you to use the M as a an argument or result type in methods.
I'm a bit puzzled about the self bit ...
Lastly the self bit is another trick that is necessary because F-bounded polymorphism was not enough in this particular case. Usually it is used with trait when traits are used as a mix-in. In such case you might want to restrict in what classes the trait can be mixed in. And at the same type it allows you to use the methods from that base type in your mixin trait.
I'd say that the particular usage in my answer is a bit unconventional but it has the same twofold effect:
When compiling OutputMessage the compiler can assume that the type will also somehow be of the type of M (whatever M is)
When compiling any sub-type compiler ensures that the constraint #1 is satisfied. For example it will not let you to do
case class SomeChild(i: Int) extends OutputMessage[SomeChild]
// this will fail because passing SomeChild breaks the restriction of self:M
case class AnotherChild(i: Int) extends OutputMessage[SomeChild]
Actually since I had to use self:M anyway, you probably can remove the F-bounded part here, living just
abstract class OutputMessage[M]() {
self: M =>
...
}
but I'd stay with it to better convey the meaning.
As SergGr already answered, you would need an F-Bounded kind of polymorphism to solve this as it is right now.
However, for these cases, I believe (note this is only my opinion) is better to use Typeclasses instead.
In your case, you only want to provide a toJson method to any value as long as they have an instance of the OFormat[T] class.
You can achieve that with this (more simple IMHO) piece of code.
object syntax {
object json {
implicit class JsonOps[T](val t: T) extends AnyVal {
def toJson(implicit: fmt: OFormat[T]): JsVal = Json.toJson(t)(fmt)
}
}
}
final case class NotifyTableChange(tableStatus: BizTable)
object NotifyTableChange {
implicit val fmt: OFormat[NotifyTableChange] = Json.format[NotifyTableChange]
}
import syntax.json._
val m = NotifyTableChange(tableStatus = ???)
val mJson = m.toJson // This works!
The JsonOps class is an Implicit Class which will provide the toJson method to any value for which there is an implicit OFormat instance in scope.
And since the companion object of the NotifyTableChange class defines such implicit, it is always in scope - more information about where does scala look for implicits in this link.
Additionally, given it is a Value Class, this extension method does not require any instantiation in runtime.
Here, you can find a more detailed discussion about F-Bounded vs Typeclasses.
No instance of play.api.libs.json.Format is available for models.AccountStatus in the implicit scope.
This is the code taken from a github page, and only class names and variable names are changed.
package models
import slick.jdbc.H2Profile._
import play.api.libs.json._
case class Account(id: Long, name: String, category: Int, status:AccountStatus)
object Account {
implicit val accountFormat = Json.format[Account]
}
sealed abstract class AccountStatus(val as:Int)
object AccountStatus{
final case object Draft extends AccountStatus(0)
final case object Active extends AccountStatus(1)
final case object Blocked extends AccountStatus(2)
final case object Defaulter extends AccountStatus(3)
implicit val columnType: BaseColumnType[AccountStatus] = MappedColumnType.base[AccountStatus,Int](AccountStatus.toInt, AccountStatus.fromInt)
private def toInt(as:AccountStatus):Int = as match {
case Draft => 0
case Active => 1
case Blocked => 2
case Defaulter => 3
}
private def fromInt(as: Int): AccountStatus = as match {
case 0 => Draft
case 1 => Active
case 2 => Blocked
case 3 => Defaulter
_ => sys.error("Out of bound AccountStatus Value.")
}
}
https://github.com/playframework/play-scala-slick-example/blob/2.6.x/app/models/Person.scala
So, this code needs to be added inside of the object AccountStatus code block since we need to use fromInt to transform an Int to an AccountStatus. This is a Reads defined for AccountStatus:
implicit object AccountStatusReads extends Reads[AccountStatus] {
def reads(jsValue: JsValue): JsResult[AccountStatus] = {
(jsValue \ "as").validate[Int].map(fromInt)
}
}
What's a Reads? It's just a trait that defines how a JsValue (the play class encapsulating JSON values) should be deserialized from JSON to some type. The trait only requires one method to be implemented, a reads method which takes in some json and returns a JsResult of some type. So you can see in the above code that we have a Reads that will look for a field in JSON called as and try to read it as an integer. From there, it will then transform it into an AccountStatus using the already defined fromInt method. So for example in the scala console you could do this:
import play.api.libs.json._
// import wherever account status is and the above reader
scala> Json.parse("""{"as":1}""").as[AccountStatus]
res0: AccountStatus = Active
This reader isn't perfect though, mainly because it's not handling the error your code will give you on out of bound numbers:
scala> Json.parse("""{"as":20}""").as[AccountStatus]
java.lang.RuntimeException: Out of bound AccountStatus Value.
at scala.sys.package$.error(package.scala:27)
at AccountStatus$.fromInt(<console>:42)
at AccountStatusReads$$anonfun$reads$1.apply(<console>:27)
at AccountStatusReads$$anonfun$reads$1.apply(<console>:27)
at play.api.libs.json.JsResult$class.map(JsResult.scala:81)
at play.api.libs.json.JsSuccess.map(JsResult.scala:9)
at AccountStatusReads$.reads(<console>:27)
at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
at play.api.libs.json.JsObject.as(JsValue.scala:166)
... 42 elided
You could handle this by making the Reads handle the error. I can show you how if you want, but first the other part of a Format is a Writes. This trait, unsurprisingly is similar to reads except it does the reverse. You're taking your class AccountStatus and creating a JsValue (JSON). So, you just have to implement the writes method.
implicit object AccountStatusWrites extends Writes[AccountStatus] {
def writes(as: AccountStatus): JsValue = {
JsObject(Seq("as" -> JsNumber(as.as)))
}
}
Then this can be used to serialize that class to JSON like so:
scala> Json.toJson(Draft)
res4: play.api.libs.json.JsValue = {"as":0}
Now, this is actually enough to get your error to go away. Why? Because Json.format[Account] is doing all the work we just did for you! But for Account. It can do this because it's a case class and has less than 22 fields. Also every field for Account has a way to be converted to and from JSON (via a Reads and Writes). Your error message was showing that Account could not have a format automatically created for it because part of it (status field) had no formatter.
Now, why do you have to do this? Because AccountStatus is not a case class, so you can't call Json.format[AccountStatus] on it. And because the subclasses of it are each objects, which have no unapply method defined for them since they're not case classes. So you have to explain to the library how to serialize and deserialize.
Since you said you're new to scala, I imagine that the concept of an implicit is still somewhat foreign. I recommend you play around with it / do some reading to get a grasp of what to do when you see that the compiler is complaining about not being able to find an implicit it needs.
Bonus round
So, you might really not want to do that work yourself, and there is a way to avoid having to do it so you can do Json.format[AccountStatus]. You see Json.format uses the apply and unapply methods to do its dirty work. In scala, these two methods are defined automatically for case classes. But there's no reason you can't define them yourself and get everything they give you for free!
So, what do apply and unapply look like type signature wise? It changes per class, but in this case apply should match Int => AccountStatus (a function that goes from an int to an AccountStatus). So it's defined like so:
def apply(i: Int): AccountStatus = fromInt(i)
and unapply is similar to the reverse of this, but it needs to return an Option[Int], so it looks like
def unapply(as: AccountStatus): Option[Int] = Option(as.as)
with both of these defined you don't need to define the reads and writes yourself and instead can just call
// this is still inside the AccountStatus object { ... }
implicit val asFormat = Json.format[AccountStatus]
and it will work in a similar fashion.
.P.S. I'm traveling today, but feel free to leave any comments if some of this doesn't make sense and I'll try to get back to you later on
Consider this example using Play's JSON API (play.api.libs.json):
case class FooJson(
// lots of other fields omitted
location: Option[LocationJson]
)
object FooJson {
implicit val writes = Json.writes[FooJson]
}
and
case class LocationJson(latitude: Double, longitude: Double)
object LocationJson {
implicit val writes = Json.writes[LocationJson]
}
If location is None, the resulting JSON won't have location field at all. This is fine and understadable. But if I wanted for some reason (say, to make my API more self-documenting), how can I explicitly output null in JSON?
{
"location": null
}
I also tried defining the field as location: LocationJson and passing option.orNull to it, but it does not work (scala.MatchError: null at play.api.libs.json.OWrites$$anon$2.writes). For non-custom types such as String or Double, this approach would produce null in JSON output.
So, while using Json.writes[FooJson] as shown above (or something equally simple, i.e. not having to write a custom Writes implementation), is there a clean way to include nulls in JSON?
What I'm asking is analogous to JsonInclude.Include.ALWAYS in the Jackson library (also Jackson's default behaviour). Similarly in Gson this would be trivial
(new GsonBuilder().serializeNulls().create()).
Play 2.4.4
Greg Methvin, a Play committer, wrote this answer to me in a related GitHub issue:
The JSON macros only support one way of encoding optional values,
which is to omit None values from the JSON. This is not a bug but
rather a limitation of the implementation. If you want to include
nulls you're unfortunately going to have to implement your own Writes.
I do think we should try to provide more configurability for the
macros though.
In this case, I'll let Play exclude this field when the value is null, even if it slightly sacrifices API consistency and self-documentability. It's still such a minor thing (in this particular API) that it doesn't warrant uglifying the code as much as a custom Writes would take for a case class with a dozen values.
I'm hoping they do make this more configurable in future Play versions.
Hello from the future.
As of Play 2.7, a fairly simple solution was introduced for automated JSON codecs. We can introduce the appropriate implicit value for JsonConfiguration in the scope for the Format/Reads/Writes. The following configuration will write nulls for empty Options instead of omitting the fields entirely.
import play.api.libs.json._
implicit val config = JsonConfiguration(optionHandlers = OptionHandlers.WritesNull)
implicit val residentWrites = Json.writes[Resident]
Reference
Here's a way to do it:
object MyWrites extends DefaultWrites{
override def OptionWrites[T](implicit fmt: Writes[T]): Writes[Option[T]] = new Writes[Option[T]] {
override def writes(o: Option[T]): JsValue = {
o match {
case Some(a) => Json.toJson(a)(fmt)
case None => JsNull
}
}
}
}
This will overwrite the default implementation which will not create an element. I used this in your sample code:
case class FooJson(
// ...
location: Option[LocationJson]
)
case class LocationJson(latitude: Double, longitude: Double)
object LocationJson {
implicit val writes = Json.writes[LocationJson]
}
implicit val fooJsonWriter: Writes[FooJson] = new Writes[FooJson] {
override def writes(o: FooJson): JsValue = {
JsObject(Seq(
"location" -> Json.toJson(o.location)
// Additional fields go here.
))
}
}
Json.toJson(FooJson(None))
And got this result res0: play.api.libs.json.JsValue = {"location":null}.
if we have null values then we have to add the option with members in case class which will resolve the issue
case class response(
name:String,
age: option[int]
)
object response {
implicit val format = Json.format[response]
}
Here the option is the answer for us. and if we are the JSON response for age is coming as null and this will handle the solution for us.
Let's assume I have a case class with the following setup:
case class Place(id:java.util.UUID, name:String)
I can write a (working!) serializer for this type as follows:
class placeSerializer extends CustomSerializer[Place]( format => (
{
case JObject(JField("id", JString(s)) :: JField("name",JString(x)) :: Nil ) =>
Place(UUID.fromString(s), x)
},
{
case x:Place =>
JObject(
JField("id", JString(x.id.toString())) ::
JField("name", JString(x.name)) :: Nil)
}
)
)
But assuming my case class eventually has a lot more fields, this could lead to me enumerating the entire structure of the object with the AST, creating something that's very verbose just to encode primitives.
json4s appears to have field serializers that can act only on specific fields, with boilerplate methods included to easily transform names and discard fields. These, however, have the following signature for their serialize and deserialize partial functions:
case class FieldSerializer[A: Manifest](
serializer: PartialFunction[(String, Any), Option[(String, Any)]] = Map(),
deserializer: PartialFunction[JField, JField] = Map()
)
Since JField (the type that representes a key -> val from the json) is its own type and not a subclass of JValue, how can I combine these two types of serializers to properly encode the id key by its name to a UUID, while maintaining the default handling of the other fields (which are primitive datatypes).
Essentially I'd like a format chain that understands the field within Place is a UUID, without having to specify AST structure for all the fields that DefaultFormats can already handle.
What I'm looking for specifically is to mimic a pattern similar to the JSONEncoder and JSONDecoder interfaces in python, which can use the key name as well as value type to determine how to handle the marshalling for the field.
There is now a UUID serializer provided in the extras package of json4s. It will most likely be available in version 3.2.11 (which has not been released as of this writing).
You'll be able to do something like this:
import org.json4s.ext.JavaTypesSerializers
implicit val json4sFormats = Serialization.formats(NoTypeHints) ++ JavaTypesSerializers.all
This was taken from the tests for this feature's PR.
The trick is to not write a serializer for your type, but for the type that you're using inside (in this case java.util.UUID)
Then you can add that serializer to the toolbox and from then any type using UUID will work exactly like types using DefaultSerializer supported fields did:
case object UUIDSerialiser extends CustomSerializer[UUID](format => (
{
case JString(s) => UUID.fromString(s)
case JNull => null
},
{
case x: UUID => JString(x.toString)
}
)
)
implicit val json4sFormats = Serialization.formats(NoTypeHints) + UUIDSerialiser
Update link to the PR
Update 2 the PR was merged, and now, in case of UUID you can use:
import org.json4s.ext.JavaTypesSerializers
implicit val json4sFormats = Serialization.formats(NoTypeHints) ++ JavaTypesSerializers.all
I need to parse some messages. The first 4 bytes of a message identify the type of message, so, using that, I can instantiate an object of the proper type. To make this an efficient operation, I thought I would create a hash map where they key is the first 4 bytes, and the value is the object constructor. I can just look up the constructor and invoke it.
After all, constructors are just functions, and there shouldn't be any problem putting functions in a map. It turns out that I am having some difficulty with this because I don't know how to express the reference to the constructor properly.
To get concrete with a simplified example, suppose we have a message base class, MsgBase, and a couple subclasses, MsgA and MsgB. If I create a companion object for each of the messages and put a factory function into it, I can make the array without any problem using those functions.
Here is a simplified sample which takes the message as a string.
class MsgBase(message: String) { }
class MsgA(message: String) extends MsgBase(message) { }
object MsgA { def makeIt(message: String): MsgA = new MsgA(message) }
and where MsgB is similar. Then I can make the map:
val cm = Map[String, (String) => MsgBase]("a" -> MsgA.makeIt, "b" -> MsgB.makeIt)
val myMsg = cm("a")("a.This is the message")
It seems like I should be able to refer to the message object constructor directly in the expression building the map, rather than using the trivial function in the companion object, but I haven't figured out any way to express that. Is there a way?
Try
"a" -> (new MsgA(_))
(all parentheses are needed).
Even if this didn't work, you could of course always define the function explicitly:
"a" -> ( (s: String) => new MsgA(s) )
For this case it would be better to use case classes, which automatically provide you functions for creating new objects.
scala> case class MsgA(message: String) extends MsgBase(message)
scala> case class MsgB(message: String) extends MsgBase(message)
So you can refer them just by name, without any syntactical overhead
scala> val m = Map("a"->MsgA, "b"->MsgB)
m: scala.collection.immutable.Map[java.lang.String,scala.runtime.AbstractFunction1[java.lang.String,Product with MsgBase]] = Map((a,<function1>), (b,<function1>))
scala> m("a")("qqq")
res1: Product with MsgBase = MsgA(qqq)
As an alternative approach you can create companion object with overrided apply method by hand. For details see Programming scala, chapter 6
val cm = Map[String, (String) => MsgBase]("a" -> (new MsgA(_)), "b" -> (new MsgB(_)))