Experimenting with Circe to serialize and deserialize Algebraic Data Types in Scala I tried out the following sample, following the doc and some samples on the web:
sealed trait StringData
case class Data2(f1: String, f2: String) extends StringData
case class Data3(f1: String, f2: String, f3: String) extends StringData
object StringData {
// ===> does not work, always picks Data2 type
implicit val decodeData: Decoder[Data] = Decoder[OptionsData].map[Data](identity).or(Decoder[TextData].map[Data](identity))
implicit val encodeData: Encoder[StringData] = Encoder.instance {
case d2 # Data2( _,_) => d2.asJson
case d3 # Data3( _, _, _) => d3.asJson
}
def toJson(s: StringData): String = s.asJson.noSpaces
def fromJson(s: String): Either[Error, StringData] = decode[StringData](s)
}
"Inheritance ADT with identical fields" should "serialize and deserialize with Circe" in {
val d2 = Data2("a", "b")
val d3 = Data3("1", "2", "3")
val jd2 = StringData.toJson(d2)
val jd3 = StringData.toJson(d3)
val d2Decoded = StringData.fromJson(jd2)
val d3Decoded = StringData.fromJson(jd3)
d2Decoded.right.get should equal(d2)
d3Decoded.right.get should equal(d3)
println("")
}
The problem is that the type of d3Decoded is always of type Data2 and not the desired Data3.
The solution I came up with was to replace the decoder with this one:
implicit val decodeData: Decoder[StringData] = Decoder.instance { c =>
c.downField("f3").as[String] match {
case m: Either[DecodingFailure, String] if m.isLeft => c.as[Data2]
case m: Either[DecodingFailure, String] if m.isRight => c.as[Data3]
}
}
This seems to me a rather ad-hoc solution. In Jackson there is the possibility to add the type to the Json. I wonder if I use Circe the right way or if this is really the way to do. Any comments very welcome.
Try using discriminator documented here:
import io.circe.generic.extras.auto._
import io.circe.generic.extras.Configuration
import io.circe.parser.decode
implicit val genDevConfig: Configuration =
Configuration.default.withDiscriminator("what_am_i")
sealed trait StringData
case class Data2(f1: String, f2: String) extends StringData
case class Data3(f1: String, f2: String, f3: String) extends StringData
decode[StringData]("""{ "f1": "foo", "f2": "bar", "f3": "qux", "what_am_i": "Data3" }""")
which outputs
res0: Either[io.circe.Error,StringData] = Right(Data3(foo,bar,qux)
where
libraryDependencies ++= Seq(
"io.circe" %% "circe-core",
"io.circe" %% "circe-generic",
"io.circe" %% "circe-parser",
"io.circe" %% "circe-generic-extras",
).map(_ % circeVersion)
Related
Consider:
import io.circe.generic.auto._, io.circe.syntax._
sealed trait Data
case class Failed(foo: String, bar: String) extends Data
case class Success(foo1:String, bar1:String) extends Data
case class Task(Foo:String, Data: Data)
val something = Task("test", Failed("1", "2"))
println(something.asJson)
This outputs:
val something: Task = Task(test,Failed(1,2))
{"Foo" : "test", "Data" : {"Failed" : {"foo" : "1","bar" : "2"}}}
But what I really want, is it to output:
{"Foo" : "test", "Data" : {"foo" : "1", "bar" : "2"}}
Effectively, I just want to delete the "Failed" block but keep everything within that block.
Build info:
val scalaVer = "2.13.8"
lazy val circeJsonSchemaVersion = "0.2.0"
lazy val circeVersion = "0.14.3"
lazy val circeOpticsVersion = "0.14.1"
"io.circe" %% "circe-json-schema" % circeJsonSchemaVersion,
"io.circe" %% "circe-core" % circeVersion,
"io.circe" %% "circe-generic" % circeVersion,
"io.circe" %% "circe-parser" % circeVersion,
"io.circe" %% "circe-literal" % circeVersion,
"io.circe" %% "circe-generic-extras" % circeVersion,
"io.circe" %% "circe-optics" % circeOpticsVersion,
I have tried using #JsonCodec but wasn't able to get it working. I looked at custom codecs but that seems like it could be a giant rabbit hole.
EDIT: Fixed bad copy/paste in output
A more generic solution recommends
import GenericDerivation.{ decodeEvent => _, encodeEvent => _ }
object ShapesDerivation {
import io.circe.shapes // "io.circe" %% "circe-shapes" % ...
import shapeless.{ Coproduct, Generic }
implicit def encodeAdtNoDiscr[A, Repr <: Coproduct](implicit
gen: Generic.Aux[A, Repr],
encodeRepr: Encoder[Repr]
): Encoder[A] = encodeRepr.contramap(gen.to)
//...
}
but this seems to be outdated.
You can try to derive encoders for non-labelled coproducts manually yourself (i.e. coproducts of the form Failed :+: Success :+: CNil rather than FieldType[Symbol ## "Failed", Failed] :+: FieldType[Symbol ## "Success", Success] :+: CNil)
import io.circe.Encoder
import shapeless.{:+:, CNil, Coproduct, Generic, Inl, Inr}
trait NoLabelCoproductEncoders {
implicit def noLabelEnc[A, Repr <: Coproduct](implicit
gen: Generic.Aux[A, Repr],
enc: NoLabelCoproductEncoder[Repr]
): Encoder[A] = enc.contramap(gen.to)
}
trait NoLabelCoproductEncoder[A <: Coproduct] extends Encoder[A]
object NoLabelCoproductEncoder {
implicit def ccons[H, T <: Coproduct](implicit
hEnc: Encoder[H],
tEnc: NoLabelCoproductEncoder[T]
): NoLabelCoproductEncoder[H :+: T] = {
case Inl(h) => hEnc(h)
case Inr(t) => tEnc(t)
}
implicit val cnil: NoLabelCoproductEncoder[CNil] = _.impossible
}
This works with semiauto
import io.circe.generic.semiauto
sealed trait Data
object Data extends NoLabelCoproductEncoders
case class Failed(foo: String, bar: String) extends Data
object Failed {
implicit val failedEnc: Encoder[Failed] = semiauto.deriveEncoder[Failed]
}
case class Success(foo1:String, bar1:String) extends Data
object Success {
implicit val successEnc: Encoder[Success] = semiauto.deriveEncoder[Success]
}
case class Task(Foo:String, Data: Data)
object Task {
implicit def taskEnc: Encoder[Task] = semiauto.deriveEncoder
}
import io.circe.syntax._
val something = Task("test", Failed("1", "2"))
something.asJson.noSpaces
// {"Foo":"test","Data":{"foo":"1","bar":"2"}}
and with auto
sealed trait Data
object Data extends NoLabelCoproductEncoders
case class Failed(foo: String, bar: String) extends Data
case class Success(foo1:String, bar1:String) extends Data
case class Task(Foo:String, Data: Data)
import io.circe.generic.auto._
import Data._ // to make noLabelEnc of higher priority for Data, otherwise it's ambiguous with Circe auto._ encoders (in LowPriorityEncoders)
import io.circe.syntax._
val something = Task("test", Failed("1", "2"))
something.asJson.noSpaces
// {"Foo":"test","Data":{"foo":"1","bar":"2"}}
I have a need to serialize a handful of case classes to only strings and ints. Meaning, if there's a nested type, it gets serialized as the stringified version of a JSON object, not a JSON object.
Example:
case class Deepest(someNum: Int)
case class Inner(superDeep: Deepest)
case class Outer(aValue: Int, aNestedValue: Inner)
Serializing an instance of Outer Would result in (or something similar)
{
"Outer": {
"aValue": 5,
"aNestedValue": "{ \"superDeep\": .... }"
}
}
Is this possible?
Using jsoniter-scala is the simplest and most efficient way to do it with Scala.
Add dependencies:
libraryDependencies ++= Seq(
// Use the %%% operator instead of %% for Scala.js
"com.github.plokhotnyuk.jsoniter-scala" %% "jsoniter-scala-core" % "2.12.0",
// Use the "provided" scope instead when the "compile-internal" scope is not supported
"com.github.plokhotnyuk.jsoniter-scala" %% "jsoniter-scala-macros" % "2.12.0" % "compile-internal"
)
Use a custom codec from this snippet:
import com.github.plokhotnyuk.jsoniter_scala.macros._
import com.github.plokhotnyuk.jsoniter_scala.core._
object Example01 {
sealed trait Message
case class Deepest(someNum: Int)
case class Inner(superDeep: Deepest)
case class Outer(aValue: Int, aNestedValue: Inner) extends Message
implicit val innerCodec: JsonValueCodec[Inner] = new JsonValueCodec[Inner] {
private val codec: JsonValueCodec[Inner] = JsonCodecMaker.make
override def decodeValue(in: JsonReader, default: Inner): Inner =
readFromStringReentrant(in.readString(null))(codec)
override def encodeValue(x: Inner, out: JsonWriter): Unit =
out.writeVal(writeToStringReentrant(x)(codec))
override def nullValue: Inner = null
}
implicit val messageCodec: JsonValueCodec[Message] =
JsonCodecMaker.make(CodecMakerConfig.withDiscriminatorFieldName(None))
def main(args: Array[String]): Unit = {
val message = readFromString[Message](
"""{
| "Outer": {
| "aValue": 5,
| "aNestedValue": "{ \"superDeep\": { \"someNum\": 1 } }"
| }
|}""".stripMargin)
println(writeToString[Message](message, WriterConfig.withIndentionStep(4)))
}
}
Expected output:
{
"Outer": {
"aValue": 5,
"aNestedValue": "{\"superDeep\":{\"someNum\":1}}"
}
}
If more than one type should be stringified you can define a function that takes a regular codec and return a codec that stringify output of original one:
def makeStringifyingCodec[A](codec: JsonValueCodec[A]): JsonValueCodec[A] =
new JsonValueCodec[A] {
override def decodeValue(in: JsonReader, default: A): A =
readFromStringReentrant(in.readString(null))(codec)
override def encodeValue(x: A, out: JsonWriter): Unit =
out.writeVal(writeToStringReentrant(x)(codec))
override def nullValue: A = null.asInstanceOf[A]
}
And use it for field types of the Outer class:
implicit val innerCodec: JsonValueCodec[Inner] =
makeStringifyingCodec(JsonCodecMaker.make[Inner])
implicit val messageCodec: JsonValueCodec[Message] =
JsonCodecMaker.make(CodecMakerConfig.withDiscriminatorFieldName(None))
I am using the Play Framework and trying to build JSON validator for a class with abstract members. Shown below, the DataSource class is the base class which I am trying to validate the format against.
// SourceTypeConfig Trait.
trait SourceTypeConfig
final case class RDBMSConfig(...) extends SourceTypeConfig
object RDBMSConfig { implicit val fmt = Json.format[RDBMSConfig] }
final case class DirectoryConfig(
path: String,
pathType: String // Local, gcloud, azure, aws, etc.
) extends SourceTypeConfig
object DirectoryConfig { implicit val fmt = Json.format[DirectoryConfig] }
// FormatConfig trait.
trait FormatConfig
final case class SQLConfig(...) extends FormatConfig
object SQLConfig { implicit val fmt = Json.format[SQLConfig]}
final case class CSVConfig(
header: String,
inferSchema: String,
delimiter: String
) extends FormatConfig
object CSVConfig { implicit val fmt = Json.format[CSVConfig]}
// DataSource base class.
case class DataSource(
name: String,
sourceType: String,
sourceTypeConfig: SourceTypeConfig,
format: String,
formatConfig: FormatConfig
)
What I am hoping to accomplish:
val input: JsValue = Json.parse(
"""
{
"name" : "test1",
"sourceType" : "directory",
"sourceTypeConfig" : {"path" : "gs://test/path", "pathType" "google"},
"format" : "csv",
"formatConfig" : {"header" : "yes", "inferSchema" : "yes", "delimiter" : "|"}
}
"""
)
val inputResult = input.validate[DataSource]
What I am struggling with is building the DataSource object and defining its reads/writes/format. I would like it to contain a match based on the sourceType and format values that direct it to point towards the associated sourceTypeConfig and formatConfig's formats so it can parse out the JSON.
Instead of building a parser at the DataSource level, I defined parsers at the SourceConfig and FormatConfig levels, similar to what is shown below.
sealed trait SourceConfig{val sourceType: String}
object SourceConfig{
implicit val fmt = new Format[SourceConfig] {
def reads(json: JsValue): JsResult[SourceConfig] = {
def from(sourceType: String, data: JsObject): JsResult[SourceConfig] = sourceType match {
case "RDBMS" => Json.fromJson[RDBMSConfig](data)(RDBMSConfig.fmt)
case "directory" => Json.fromJson[DirectoryConfig](data)(DirectoryConfig.fmt)
case _ => JsError(s"Unknown source type: '$sourceType'")
}
for {
sourceType <- (json \ "sourceType").validate[String]
data <- json.validate[JsObject]
result <- from(sourceType, data)
} yield result
}
def writes(source: SourceConfig): JsValue =
source match {
case b: RDBMSConfig => Json.toJson(b)(RDBMSConfig.fmt)
case b: DirectoryConfig => Json.toJson(b)(DirectoryConfig.fmt)
}
}
}
Then, DataSource could be simply defined as:
object DataSource { implicit val fmt = Json.format[DataSource] }
Another option is to use play-json-derived-codecs library:
libraryDependencies += "org.julienrf" %% "play-json-derived-codecs" % "4.0.0"
import julienrf.json.derived.flat
implicit val format1: OFormat[RDBMSConfig] = Json.format[RDBMSConfig]
implicit val format2: OFormat[DirectoryConfig] = Json.format[DirectoryConfig]
implicit val format3: OFormat[SourceTypeConfig] = flat.oformat((__ \ "sourceType").format[String])
I am not much familier with spray json, but I have to convert the below json into Array[myTest]
Below is the code, but it doesnt work. It throws the following errors: How do I fix them?
Error:(19, 54) Cannot find JsonReader or JsonFormat type class for Array[A$A61.this.myTest]
lazy val converted= trainingDataRef.toJson.convertTo[Array[myTest]]
^
Error:(19, 54) not enough arguments for method convertTo: (implicit evidence$1: spray.json.JsonReader[Array[A$A61.this.myTest]])Array[A$A61.this.myTest].
Unspecified value parameter evidence$1.
lazy val converted= trainingDataRef.toJson.convertTo[Array[myTest]]
^
Error:(10, 61) could not find implicit value for evidence parameter of type spray.json.DefaultJsonProtocol.JF[Map[String,Any]]
implicit val format: RootJsonFormat[myTest] = jsonFormat3(myTest.apply)
^
Code: ^
import spray.json.DefaultJsonProtocol._
import spray.json._
case class myTest (
id: String,
classDetails: Map[String, Any],
school: Map[String, Any])
object myTest {
implicit val format: RootJsonFormat[myTest] = jsonFormat3(myTest.apply)
}
val trainingDataRef = """[{"id":"my-id","classDetails":{"sec":"2","teacher":"John"},"school":{"name":"newschool"}}]"""
println(trainingDataRef.getClass)
val converted= trainingDataRef.toJson.convertTo[Array[myTest]]
println(converted)
spray-json has good documentation, try take a look there. Basically, you have to define your case classes and implement JsonFormat for them:
import spray.json.DefaultJsonProtocol._
import spray.json._
case class ClassDetails(sec: String, teacher: String)
object ClassDetails {
implicit val format: RootJsonFormat[ClassDetails] = jsonFormat2(ClassDetails.apply)
}
case class School(name: String)
object School {
implicit val format: RootJsonFormat[School] = jsonFormat1(School.apply)
}
case class ClassInfo
(
id: String,
classDetails: ClassDetails,
school: School
)
object ClassInfo {
implicit object ClassInfoFormat extends RootJsonFormat[ClassInfo] {
def write(c: ClassInfo): JsValue = JsObject(
"id" -> JsString(c.id),
"classDetails" -> c.classDetails.toJson,
"school" -> c.school.toJson
)
def read(value: JsValue): ClassInfo = {
value.asJsObject.getFields("id", "classDetails", "school") match {
case Seq(JsString(name), details, school) =>
new ClassInfo(name, details.convertTo[ClassDetails], school.convertTo[School])
case _ => throw new DeserializationException("ClassInfo expected")
}
}
}
}
val json = """[{"id":"my-id","classDetails":{"sec":"2","teacher":"John"},"school":{"name":"newschool"}}]"""
// JSON string to case classes
val classInfos = json.parseJson.convertTo[Seq[ClassInfo]]
classInfos.zipWithIndex.foreach { case (c, idx) =>
println(s"$idx => $c")
}
println
// Seq[ClassInfo] to JSON
println(s"$classInfos: ")
println(classInfos.toJson.prettyPrint)
I'm trying to deserialize a JsArray into a List[T] in a playframework application using Scala. After some research I found this method which is supposed to do the needed work:
/**
* Deserializer for List[T] types.
*/
implicit def listReads[T](implicit fmt: Reads[T]): Reads[List[T]] = new Reads[List[T]] {
def reads(json: JsValue) = json match {
case JsArray(ts) => ts.map(t => fromJson(t)(fmt)).toList
case _ => throw new RuntimeException("List expected")
}
}
The problem is that I didn't know how to use it. Any help is welcome.
Here's a quick example:
scala> import play.api.libs.json._
import play.api.libs.json._
scala> Json.toJson(List(1, 2, 3)).as[List[Int]]
res0: List[Int] = List(1, 2, 3)
And if you have a custom type with a Format instance:
case class Foo(i: Int, x: String)
implicit object fooFormat extends Format[Foo] {
def reads(json: JsValue) = Foo(
(json \ "i").as[Int],
(json \ "x").as[String]
)
def writes(foo: Foo) = JsObject(Seq(
"i" -> JsNumber(foo.i),
"x" -> JsString(foo.x)
))
}
It still works:
scala> val foos = Foo(1, "a") :: Foo(2, "bb") :: Nil
foos: List[Foo] = List(Foo(1,a), Foo(2,bb))
scala> val json = Json.toJson(foos)
json: play.api.libs.json.JsValue = [{"i":1,"x":"a"},{"i":2,"x":"bb"}]
scala> json.as[List[Foo]]
res1: List[Foo] = List(Foo(1,a), Foo(2,bb))
This approach would also work if your custom type had a xs: List[String] member, for example: you'd just use (json \ "xs").as[List[String]] in your reads method and Json.toJson(foo.xs) in your writes.