I have used Circe previously for case class serialization / deserialization, and love how it can be used without the boilerplate code required by other Scala JSON libraries, but I'm running into an issue now I'm not sure how to resolve. I have an ADT (a sealed trait with several case class instances) that I would like to treat (from my Akka Http Service, using akka-http-json) generically (ie, return a List[Foo], where Foo is the trait-type), but when I do so using Circe's auto-deriviation (via Shapeless), it serializes the instances using the specific case class name as a 'discriminator' (eg, if my List[Foo] contains instances of Foo1, then each element in the resulting serialized list will have the key Foo1). I would like to eliminate the type name as a discriminator (ie, so that instead of having each element in the sequence prefixed with the type name-- eg, "Foo1": {"id : "1", name : "First",...}, I just want to serialize the case class instances to contain the fields of the case class: eg, {"id":"1,"name:"First",...}...Essentially, I'd like to eliminate the type name keys (I don't want the front-end to have to know what concrete case class each element belongs to on the back-end).All elements in the list to be serialized will be of the same concrete-type, all of which would be subtypes of my ADT (trait) type. I believe this can be done using Circe's semi-auto derivation, though I haven't had a chance to figure out exactly how. Basically, I would like to use as much of Circe's auto-derivation as possible, but eliminate outer-level class names from appearing in the resulting JSON. Any help / suggestions would be very much appreciated! Thanks!
you can do it following the instruction in the doc: https://circe.github.io/circe/codecs/adt.html
import cats.syntax.functor._
import io.circe.{ Decoder, Encoder }, io.circe.generic.auto._
import io.circe.syntax._
object GenericDerivation {
implicit val encodeEvent: Encoder[Event] = Encoder.instance {
case foo # Foo(_) => foo.asJson
case bar # Bar(_) => bar.asJson
case baz # Baz(_) => baz.asJson
case qux # Qux(_) => qux.asJson
}
implicit val decodeEvent: Decoder[Event] =
List[Decoder[Event]](
Decoder[Foo].widen,
Decoder[Bar].widen,
Decoder[Baz].widen,
Decoder[Qux].widen
).reduceLeft(_ or _)
}
import GenericDerivation._
import io.circe.parser.decode
decode[Event]("""{ "i": 1000 }""")
// res0: Either[io.circe.Error,Event] = Right(Foo(1000))
(Foo(100): Event).asJson.noSpaces
// res1: String = {"i":100}
This may not be the best answer, but after some more searching this is what I've been able to find. Instead of having the class name as a key in the Json produced, it can be serialized as a field as following:
implicit val genDevConfig: Configuration = Configuration.default.withDescriminator("type")
(you can use whatever field name here you'd like; Travis Brown's previous example for a similar issue used a field named what_am_i). So my apologies-- I do not yet know if there is a canonical or widely accepted solution to this problem, especially one that will easily work with Akka Http, using libraries such as akka-http-json, where I still seem to be encountering some issues, though I'm sure I'm probably overlooking something obvious! Anyway, my apologies for asking a question that seems to come up repeatedly!
Related
I'm trying to construct a trait and an abstract class to subtype by messages (In an Akka play environment) so I can easily convert them to Json.
What have done so far:
abstract class OutputMessage(val companion: OutputMessageCompanion[OutputMessage]) {
def toJson: JsValue = Json.toJson(this)(companion.fmt)
}
trait OutputMessageCompanion[OT] {
implicit val fmt: OFormat[OT]
}
Problem is, when I'm trying to implement the mentioned interfaces as follows:
case class NotifyTableChange(tableStatus: BizTable) extends OutputMessage(NotifyTableChange)
object NotifyTableChange extends OutputMessageCompanion[NotifyTableChange] {
override implicit val fmt: OFormat[NotifyTableChange] = Json.format[NotifyTableChange]
}
I get this error from Intellij:
Type mismatch, expected: OutputMessageCompanion[OutputMessage], actual: NotifyTableChange.type
I'm kinda new to Scala generics - so help with some explanations would be much appreciated.
P.S I'm open for any more generic solutions than the one mentioned.
The goal is, when getting any subtype of OutputMessage - to easily convert it to Json.
The compiler says that your companion is defined over the OutputMessage as the generic parameter rather than some specific subtype. To work this around you want to use a trick known as F-bound generic. Also I don't like the idea of storing that companion object as a val in each message (after all you don't want it serialized, do you?). Defining it as a def is IMHO much better trade-off. The code would go like this (companions stays the same):
abstract class OutputMessage[M <: OutputMessage[M]]() {
self: M => // required to match Json.toJson signature
protected def companion: OutputMessageCompanion[M]
def toJson: JsValue = Json.toJson(this)(companion.fmt)
}
case class NotifyTableChange(tableStatus: BizTable) extends OutputMessage[NotifyTableChange] {
override protected def companion: OutputMessageCompanion[NotifyTableChange] = NotifyTableChange
}
You may also see standard Scala collections for an implementation of the same approach.
But if all you need the companion for is to encode with JSON format, you can get rid of it like this:
abstract class OutputMessage[M <: OutputMessage[M]]() {
self: M => // required to match Json.toJson signature
implicit protected def fmt: OFormat[M]
def toJson: JsValue = Json.toJson(this)
}
case class NotifyTableChange(tableStatus: BizTable) extends OutputMessage[NotifyTableChange] {
override implicit protected def fmt: OFormat[NotifyTableChange] = Json.format[NotifyTableChange]
}
Obviously is you also want to decode from JSON you still need a companion object anyway.
Answers to the comments
Referring the companion through a def - means that is a "method", thus defined once for all the instances of the subtype (and doesn't gets serialized)?
Everything you declare with val gets a field stored in the object (instance of the class). By default serializers trying to serialize all the fields. Usually there is some way to say that some fields should be ignored (like some #IgnoreAnnotation). Also it means that you'll have one more pointer/reference in each object which uses memory for no good reason, this might or might not be an issue for you. Declaring it as def gets a method so you can have just one object stored in some "static" place like companion object or build it on demand every time.
I'm kinda new to Scala, and I've peeked up the habit to put the format inside the companion object, would you recommend/refer to some source, about how to decide where is best to put your methods?
Scala is an unusual language and there is no direct mapping the covers all the use cases of the object concept in other languages. As a first rule of thumb there are two main usages for object:
Something where you would use static in other languages, i.e. a container for static methods, constants and static variables (although variables are discouraged, especially static in Scala)
Implementation of the singleton pattern.
By f-bound generic - do you mean the lower bound of the M being OutputMessage[M] (btw why is it ok using M twice in the same expr. ?)
Unfortunately wiki provides only a basic description. The whole idea of the F-bounded polymorphism is to be able to access to the type of the sub-class in the type of a base class in some generic manner. Usually A <: B constraint means that A should be a subtype of B. Here with M <: OutputMessage[M], it means that M should be a sub-type of the OutputMessage[M] which can easily be satisfied only by declaring the child class (there are other non-easy ways to satisfy that) as:
class Child extends OutputMessage[Child}
Such trick allows you to use the M as a an argument or result type in methods.
I'm a bit puzzled about the self bit ...
Lastly the self bit is another trick that is necessary because F-bounded polymorphism was not enough in this particular case. Usually it is used with trait when traits are used as a mix-in. In such case you might want to restrict in what classes the trait can be mixed in. And at the same type it allows you to use the methods from that base type in your mixin trait.
I'd say that the particular usage in my answer is a bit unconventional but it has the same twofold effect:
When compiling OutputMessage the compiler can assume that the type will also somehow be of the type of M (whatever M is)
When compiling any sub-type compiler ensures that the constraint #1 is satisfied. For example it will not let you to do
case class SomeChild(i: Int) extends OutputMessage[SomeChild]
// this will fail because passing SomeChild breaks the restriction of self:M
case class AnotherChild(i: Int) extends OutputMessage[SomeChild]
Actually since I had to use self:M anyway, you probably can remove the F-bounded part here, living just
abstract class OutputMessage[M]() {
self: M =>
...
}
but I'd stay with it to better convey the meaning.
As SergGr already answered, you would need an F-Bounded kind of polymorphism to solve this as it is right now.
However, for these cases, I believe (note this is only my opinion) is better to use Typeclasses instead.
In your case, you only want to provide a toJson method to any value as long as they have an instance of the OFormat[T] class.
You can achieve that with this (more simple IMHO) piece of code.
object syntax {
object json {
implicit class JsonOps[T](val t: T) extends AnyVal {
def toJson(implicit: fmt: OFormat[T]): JsVal = Json.toJson(t)(fmt)
}
}
}
final case class NotifyTableChange(tableStatus: BizTable)
object NotifyTableChange {
implicit val fmt: OFormat[NotifyTableChange] = Json.format[NotifyTableChange]
}
import syntax.json._
val m = NotifyTableChange(tableStatus = ???)
val mJson = m.toJson // This works!
The JsonOps class is an Implicit Class which will provide the toJson method to any value for which there is an implicit OFormat instance in scope.
And since the companion object of the NotifyTableChange class defines such implicit, it is always in scope - more information about where does scala look for implicits in this link.
Additionally, given it is a Value Class, this extension method does not require any instantiation in runtime.
Here, you can find a more detailed discussion about F-Bounded vs Typeclasses.
I have HTTP client written in Scala that uses json4s/jackson to serialize and deserialize HTTP payloads. For now I was using only Scala case classes as model and everything was working fine, but now I have to communicate with third party service. They provided me with their own model but its written in Java, so now I need to deserialize jsons also to Java classes. It seams to work fine with simple classes but when class contains collections like Lists or Maps json4s has problems and sets all such fields to null.
Is there any way to handle such cases? Maybe I should use different formats (I'm using DefaultFormats + few custom ones). Example of problem with test:
import org.json4s.DefaultFormats
import org.json4s.jackson.Serialization.read
import org.scalatest.{FlatSpec, Matchers}
class JavaListTest extends FlatSpec with Matchers{
implicit val formats = DefaultFormats
"Java List" should "be deserialized properly" in {
val input = """{"list":["a", "b", "c"]}"""
val output = read[ObjectWithList](input)
output.list.size() shouldBe 3
}
}
And sample Java class:
import java.util.List;
public class ObjectWithList {
List<String> list;
}
I have also noticed that when I'll try to deserialize to Scala case class that contains java.util.List[String] type of field I'll get an exception of type: org.json4s.package$MappingException: Expected collection but got List[String]
Key for solving your issue, is composition of formatters. Basically you want to define JList formatter as list formatter composed with toJList function.
Unfortunately, json4s Formatters are extremely difficult to compose, so I used the Readers for you to get an idea. I also simplified an example, to having only java list:
import DefaultReaders._
import scala.collection.JavaConverters._
implicit def javaListReader[A: Reader]: Reader[java.util.List[A]] = new Reader[util.List[A]] {
override def read(value: JValue) = DefaultReaders.traversableReader[List, A].read(value).asJava
}
val input = """["a", "b", "c"]"""
val output = Formats.read[java.util.List[String]](parse(input))
To my knowledge json4s readers will not work with java classes out of the box, so you might either need to implement the Serializer[JList[_]] the same way, or mirror your java classes with case classes and use them inside your domain.
P.S.
Highly recommend you to switch to circe or argonaut, then you will forget about the most problems with jsons.
No instance of play.api.libs.json.Format is available for models.AccountStatus in the implicit scope.
This is the code taken from a github page, and only class names and variable names are changed.
package models
import slick.jdbc.H2Profile._
import play.api.libs.json._
case class Account(id: Long, name: String, category: Int, status:AccountStatus)
object Account {
implicit val accountFormat = Json.format[Account]
}
sealed abstract class AccountStatus(val as:Int)
object AccountStatus{
final case object Draft extends AccountStatus(0)
final case object Active extends AccountStatus(1)
final case object Blocked extends AccountStatus(2)
final case object Defaulter extends AccountStatus(3)
implicit val columnType: BaseColumnType[AccountStatus] = MappedColumnType.base[AccountStatus,Int](AccountStatus.toInt, AccountStatus.fromInt)
private def toInt(as:AccountStatus):Int = as match {
case Draft => 0
case Active => 1
case Blocked => 2
case Defaulter => 3
}
private def fromInt(as: Int): AccountStatus = as match {
case 0 => Draft
case 1 => Active
case 2 => Blocked
case 3 => Defaulter
_ => sys.error("Out of bound AccountStatus Value.")
}
}
https://github.com/playframework/play-scala-slick-example/blob/2.6.x/app/models/Person.scala
So, this code needs to be added inside of the object AccountStatus code block since we need to use fromInt to transform an Int to an AccountStatus. This is a Reads defined for AccountStatus:
implicit object AccountStatusReads extends Reads[AccountStatus] {
def reads(jsValue: JsValue): JsResult[AccountStatus] = {
(jsValue \ "as").validate[Int].map(fromInt)
}
}
What's a Reads? It's just a trait that defines how a JsValue (the play class encapsulating JSON values) should be deserialized from JSON to some type. The trait only requires one method to be implemented, a reads method which takes in some json and returns a JsResult of some type. So you can see in the above code that we have a Reads that will look for a field in JSON called as and try to read it as an integer. From there, it will then transform it into an AccountStatus using the already defined fromInt method. So for example in the scala console you could do this:
import play.api.libs.json._
// import wherever account status is and the above reader
scala> Json.parse("""{"as":1}""").as[AccountStatus]
res0: AccountStatus = Active
This reader isn't perfect though, mainly because it's not handling the error your code will give you on out of bound numbers:
scala> Json.parse("""{"as":20}""").as[AccountStatus]
java.lang.RuntimeException: Out of bound AccountStatus Value.
at scala.sys.package$.error(package.scala:27)
at AccountStatus$.fromInt(<console>:42)
at AccountStatusReads$$anonfun$reads$1.apply(<console>:27)
at AccountStatusReads$$anonfun$reads$1.apply(<console>:27)
at play.api.libs.json.JsResult$class.map(JsResult.scala:81)
at play.api.libs.json.JsSuccess.map(JsResult.scala:9)
at AccountStatusReads$.reads(<console>:27)
at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
at play.api.libs.json.JsObject.as(JsValue.scala:166)
... 42 elided
You could handle this by making the Reads handle the error. I can show you how if you want, but first the other part of a Format is a Writes. This trait, unsurprisingly is similar to reads except it does the reverse. You're taking your class AccountStatus and creating a JsValue (JSON). So, you just have to implement the writes method.
implicit object AccountStatusWrites extends Writes[AccountStatus] {
def writes(as: AccountStatus): JsValue = {
JsObject(Seq("as" -> JsNumber(as.as)))
}
}
Then this can be used to serialize that class to JSON like so:
scala> Json.toJson(Draft)
res4: play.api.libs.json.JsValue = {"as":0}
Now, this is actually enough to get your error to go away. Why? Because Json.format[Account] is doing all the work we just did for you! But for Account. It can do this because it's a case class and has less than 22 fields. Also every field for Account has a way to be converted to and from JSON (via a Reads and Writes). Your error message was showing that Account could not have a format automatically created for it because part of it (status field) had no formatter.
Now, why do you have to do this? Because AccountStatus is not a case class, so you can't call Json.format[AccountStatus] on it. And because the subclasses of it are each objects, which have no unapply method defined for them since they're not case classes. So you have to explain to the library how to serialize and deserialize.
Since you said you're new to scala, I imagine that the concept of an implicit is still somewhat foreign. I recommend you play around with it / do some reading to get a grasp of what to do when you see that the compiler is complaining about not being able to find an implicit it needs.
Bonus round
So, you might really not want to do that work yourself, and there is a way to avoid having to do it so you can do Json.format[AccountStatus]. You see Json.format uses the apply and unapply methods to do its dirty work. In scala, these two methods are defined automatically for case classes. But there's no reason you can't define them yourself and get everything they give you for free!
So, what do apply and unapply look like type signature wise? It changes per class, but in this case apply should match Int => AccountStatus (a function that goes from an int to an AccountStatus). So it's defined like so:
def apply(i: Int): AccountStatus = fromInt(i)
and unapply is similar to the reverse of this, but it needs to return an Option[Int], so it looks like
def unapply(as: AccountStatus): Option[Int] = Option(as.as)
with both of these defined you don't need to define the reads and writes yourself and instead can just call
// this is still inside the AccountStatus object { ... }
implicit val asFormat = Json.format[AccountStatus]
and it will work in a similar fashion.
.P.S. I'm traveling today, but feel free to leave any comments if some of this doesn't make sense and I'll try to get back to you later on
Is there any already available library that can convert JSON string (most probably more than 1 rows of data) to CSV file.
I googled a lot for any such libraries in Scala, but I could find none.
What I am required to do is retrieve Data from a DB source, the result-set are in JSON format, and convert them into CSV.
Before what I did was converted the JSON into relevant Seq[case-class] and tried to used libraries like:
Scala-csv (tototoshi)
Might-csv
But these didn't prove much useful when in case of case class containing deep hierarchies.
Any Suggestions??
product-collections will convert a Seq[case class] to csv.
case class Foo(a:Int,b:String)
Seq(Foo(1,"aa"),Foo(2,"bb")).csvIterator.mkString("\n")
res27: String =
1,"aa"
2,"bb"
To deserialize json I'd probably use scala pickling.
"Deep hierarchies" are likely to be problematic.
Not entirely sure what you mean by deep hierarchies. Something like this?
case class Baz(i: Int)
case class Bar(baz: Baz)
case class Foo(bar: Bar)
In which case kantan.csv can answer part of the equation: turning deep case class hierarchies into CSV data. It's fairly trivial, provided you don't mind a shapeless dependency:
import kantan.csv.ops._
import kantan.csv.generic.codecs._
val fs: List[Foo] = ???
fs.foldLeft(new File("output.csv").asCsvWriter[Foo])(_ write _).close
If you have an issue with shapeless, you can provide encoders for your case classes yourself:
import kantan.csv._
implicit val bazCodec = Codec.caseCodec1(Baz.apply, Baz.unapply)(0)
implicit val barCodec = Codec.caseCodec1(Bar.apply, Bar.unapply)(0)
implicit val fooCodec = Codec.caseCodec1(Foo.apply, Foo.unapply)(0)
A bit more boilerplatey, but still acceptable, I think.
I'm scala newbie and come from a Ruby background so and am having trouble rendering json response in my web service for which I use scalatra, mongodb with liftweb mongo record and argonaut for JSon serialisation and deserialisation.
However based on the examples given at http://argonaut.io/ I'm unable to figure out how this would work when using the net.liftweb.mongo.record library.
On compiling this i get a error which says a type mismatch. The error description follows the code snippet.
package firstscalatraapp
import org.scalatra
import net.liftweb.mongodb._
import net.liftweb.mongodb.record.MongoRecord
import net.liftweb.mongodb.record.field.ObjectIdPk
import net.liftweb.record.field.StringField
import net.liftweb.record.field.IntField
import net.liftweb.record.field.PasswordField
import net.liftweb.record.field.DateTimeField
import net.liftweb.mongodb.record.MongoMetaRecord
import argonaut._
import Argonaut._
case class Person private extends MongoRecord[Person] with ObjectIdPk[Person] {
def meta = Person
object age extends IntField(this, 3)
object name extends StringField(this, 29)
object created_at extends DateTimeField(this)
object password extends PasswordField(this)
}
object Person extends Person with MongoMetaRecord[Person] {
implicit def PersonCodecJson: CodecJson[Person] =
casecodec3(Person.apply, Person.unapply)("name", "age", "things")
}
The Error i get is
[error] found : () => firstscalatraapp.Person
[error] required: (?, ?, ?) => ?
[error] casecodec3(Person.apply, Person.unapply)("name", "age", "things")
[error] ^
[error] one error found
[error] (compile:compile) Compilation failed
which seems logical because the constructor does not accept any parameters and the mongo library seems to be generating the val for the fields that i need for the class (I still don't fully understand what the lift mongo wrapper does yet).
So how do i define the implicit to be able to find serialise an object of type person.
Also how do I define serialisation capabilities when i'm dealing with collections. For instance when I have a List[Person].
Thanks in advance. I would really appreciate any help i can get on this.
I'm just about to start using Argonaut so I'm no expert on that but with that said your initial problem seems obvious.
casecodec3 needs a constructor and a deconstructor for the class you're defining the codec for. In the examples of Argonaut they're using case classes and these have automatically generated companion objects with apply/unapply for the fields defined. Which for casecodec3 needs to be 3. In your case, the case class is of zero-arity - you have no case class fields at all. The fields of the record are defined as inner objects with their own apply-methods (very imperative stuff). That's just the way lifts records are defined. So your apply method is just () => Person.
casecodec3 wants a function from a 3-tuple to Person and from Person to a 3-tuple. I would suggest skipping the case definition if you're going to use lift record. And create functions on the side instead. Something like:
object Person extends Person with MongoMetaRecord[Person] {
implicit def PersonCodecJson: CodecJson[Person] =
casecodec3(parse, serialize)("name", "age", "things")
// Something like
def parse(name: String, age: Int, things: Something) = {
val p = Person.createRecord
p.name(name)
...
}
def serialize(p: Person) = (p.name.get, p.age.get, p.things.get)
}
As for your other questions I think you can head back to argonaut.io again. Their documentation seems quite alright - maybe it was worse when you posted this question as it is kind of old?
I'm going to try to replace all my serialization from lift-json to argonaut right now so if you're still stuck (probably not) I might be able to answer better in a bit.