I have a Scala Map keyed by a type that itself needs serialising to JSON. Because of the nature of JSON that requires key names for objects to be strings, a simple mapping is not directly possible.
The work around I wish to implement is to convert the Map to a Set before serialising to JSON, and then from a Set back to a Map after deserialising.
I am aware of other methods using key serialisers on specific types, e.g. Serialising Map with Jackson, however, I require a solution that applies to arbitrary key types and in this regard, conversion to Set and back again looks to me like the best option.
I've had some success serialising to a Set with a wrapper object by modifying MapSerializerModule.scala from jackson-module-scala below, but I'm not familiar enough with the Jackson internals to get that JSON to deserialise back into the Map I started with.
I should add that I control both the serialisation and deserialisation side, so what the JSON looks like is not significant.
case class Wrapper[K, V](
value: Set[(K, V)]
)
class MapConverter[K, V](inputType: JavaType, config: SerializationConfig)
extends StdConverter[Map[K, V], Wrapper[K, V]] {
def convert(value: Map[K, V]): Wrapper[K, V] = {
val set = value.toSet
Wrapper(set)
}
override def getInputType(factory: TypeFactory) = inputType
override def getOutputType(factory: TypeFactory) =
factory.constructReferenceType(classOf[Wrapper[_, _]], inputType.getContentType)
.withTypeHandler(inputType.getTypeHandler)
.withValueHandler(inputType.getValueHandler)
}
object MapSerializerResolver extends Serializers.Base {
val MAP = classOf[Map[_, _]]
override def findMapLikeSerializer(
config: SerializationConfig,
typ: MapLikeType,
beanDesc: BeanDescription,
keySerializer: JsonSerializer[AnyRef],
elementTypeSerializer: TypeSerializer,
elementValueSerializer: JsonSerializer[AnyRef]): JsonSerializer[_] = {
val rawClass = typ.getRawClass
if (!MAP.isAssignableFrom(rawClass)) null
else new StdDelegatingSerializer(new MapConverter(typ, config))
}
}
object Main {
def main(args: Array[String]): Unit = {
val objMap = Map(
new Key("k1", "k2") -> "k1k2",
new Key("k2", "k3") -> "k2k3")
val om = new ObjectMapper()
om.registerModule(DefaultScalaModule)
om.registerModule(ConverterModule)
val res = om.writeValueAsString(objMap)
println(res)
}
}
I managed to find the solution:
case class MapWrapper[K, V](
wrappedMap: Set[MapEntry[K, V]]
)
case class MapEntry[K, V](
key: K,
value: V
)
object MapConverter extends SimpleModule {
addSerializer(classOf[Map[_, _]], new StdDelegatingSerializer(new StdConverter[Map[_, _], MapWrapper[_, _]] {
def convert(inMap: Map[_, _]): MapWrapper[_, _] = MapWrapper(inMap map { case (k, v) => MapEntry(k, v) } toSet)
}))
addDeserializer(classOf[Map[_, _]], new StdDelegatingDeserializer(new StdConverter[MapWrapper[_, _], Map[_, _]] {
def convert(mapWrapper: MapWrapper[_, _]): Map[_, _] = mapWrapper.wrappedMap map { case MapEntry(k, v) => (k, v) } toMap
}))
}
class MapKey(
val k1: String,
val k2: String
) {
override def toString: String = s"MapKey($k1, $k2) (str)"
}
object Main {
def main(args: Array[String]): Unit = {
val objMap = Map(
new MapKey("k1", "k2") -> "k1k2",
new MapKey("k2", "k3") -> "k2k3")
val om = setupObjectMapper
val jsonMap = om.writeValueAsString(objMap)
val deserMap = om.readValue(jsonMap, classOf[Map[_, _]])
}
private def setupObjectMapper = {
val typeResolverBuilder =
new DefaultTypeResolverBuilder(ObjectMapper.DefaultTyping.NON_FINAL) {
init(JsonTypeInfo.Id.CLASS, null)
inclusion(JsonTypeInfo.As.WRAPPER_OBJECT)
typeProperty("#CLASS")
override def useForType(t: JavaType): Boolean = !t.isContainerType && super.useForType(t)
}
val om = new ObjectMapper()
om.registerModule(DefaultScalaModule)
om.registerModule(MapConverter)
om.setDefaultTyping(typeResolverBuilder)
om
}
}
Interestingly, if the key type is a case class, the MapConverter is not necessary since the case class can be reconstituted from the string representation.
In my case, I had a nested Map. This required a small addition to the conversion back into a map:
addDeserializer(classOf[Map[_, _]], new StdDelegatingDeserializer(new StdConverter[MapWrapper[_, _], Map[_, _]] {
def convert(mapWrapper: MapWrapper[_, _]): Map[_, _] = {
mapWrapper.wrappedMap.map { case MapEntry(k, v) => {
v match {
case wm: MapWrapper[_, _] => (k, convert(wm))
case _ => (k, v)
}
}}.toMap
}
}))
Related
I am trying to add a retry logic for JSON conversion. When converting an object to json, I am retrying for 3 times if there is any exception. I am doing :
var mapper = new ObjectMapper() with ScalaObjectMapper
intializeMapper( )
def intializeMapper() = {
// jackson library does not support seralization and deserialization of
// of scala classes like List and Map, this is needed to support it
mapper.registerModule( DefaultScalaModule )
// enables parsing of NaN. Enabling it here as JsonUtil class currently in
// use supports it.
mapper.configure(JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS, true )
mapper.setSerializationInclusion(Include.NON_NULL)
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
}
def getPersonRDD(result: DataFrame): RDD[(String, String)] = {
val finalValue = result.rdd.map({
r =>
val customerId = r.getAs[String](CUSTOMER_ID)
val itemId = r.getAs[Map[String, Int]](ITEM_ID)
val itemName = r.getAs[Map[String, Int]](ITEM_NAME)
val person = Person(itemId, itemName)
val jsonString = toJson(person)
(customerId, jsonString)
})
return finalValue
}
def fromJson(json: String, clazz: Class[_]) = {
mapper.readValue(json, clazz)
}
def toJson(value: Any): String = {
var jsonString: String = " "
jsonString = mapper.writeValueAsString(value)
try {
fromJson(jsonString, clazz)
return jsonString
} catch {
case Exception => {
publishMetrics(PARSING_EXCEPTION, 1.0)
val result = util.Try(retry() {
jsonString = mapper.writeValueAsString(value)
val features = fromJson(jsonString, clazz)
})
result match {
case util.Success(value) => jsonString
case util.Failure(error) => {
log.error("Error while parsing JSON " + jsonString)
return jsonString
}
}
}
}
}
// Returning T, throwing the exception on failure
#annotation.tailrec
def retry[T](n: Int = 3)(fn: => T): T = {
util.Try {
fn
} match {
case util.Success(x) => x
case _ if n > 1 => retry(n - 1)(fn)
case util.Failure(e) => throw e
}
}
case class Person(itemId: Map[String, Int], itemName: Map[String, Int]) extends Serializable
Is this correct ? I am new to Scala. Can someone suggest me if there is any better way for achieving this ? Is there predefined retry logic available in Scala ? The reason I am trying to add retry logic for JSON conversion is due to Jackson version I use(which I can't change for now), sometimes my writeValueAsString results in incomplete JSON.
You retry function seems correct. The only flaw I can think of is that if you expect something would fail it's better just make the return type Try[T], so you can handle it outside in the scala way.
Here is one of my implementation:
def retry[T](n: Int)(block: => T): Try[T] = {
val stream = Stream.fill(n)(Try(block))
stream find (_.isSuccess) getOrElse stream.head
}
I am not much familier with spray json, but I have to convert the below json into Array[myTest]
Below is the code, but it doesnt work. It throws the following errors: How do I fix them?
Error:(19, 54) Cannot find JsonReader or JsonFormat type class for Array[A$A61.this.myTest]
lazy val converted= trainingDataRef.toJson.convertTo[Array[myTest]]
^
Error:(19, 54) not enough arguments for method convertTo: (implicit evidence$1: spray.json.JsonReader[Array[A$A61.this.myTest]])Array[A$A61.this.myTest].
Unspecified value parameter evidence$1.
lazy val converted= trainingDataRef.toJson.convertTo[Array[myTest]]
^
Error:(10, 61) could not find implicit value for evidence parameter of type spray.json.DefaultJsonProtocol.JF[Map[String,Any]]
implicit val format: RootJsonFormat[myTest] = jsonFormat3(myTest.apply)
^
Code: ^
import spray.json.DefaultJsonProtocol._
import spray.json._
case class myTest (
id: String,
classDetails: Map[String, Any],
school: Map[String, Any])
object myTest {
implicit val format: RootJsonFormat[myTest] = jsonFormat3(myTest.apply)
}
val trainingDataRef = """[{"id":"my-id","classDetails":{"sec":"2","teacher":"John"},"school":{"name":"newschool"}}]"""
println(trainingDataRef.getClass)
val converted= trainingDataRef.toJson.convertTo[Array[myTest]]
println(converted)
spray-json has good documentation, try take a look there. Basically, you have to define your case classes and implement JsonFormat for them:
import spray.json.DefaultJsonProtocol._
import spray.json._
case class ClassDetails(sec: String, teacher: String)
object ClassDetails {
implicit val format: RootJsonFormat[ClassDetails] = jsonFormat2(ClassDetails.apply)
}
case class School(name: String)
object School {
implicit val format: RootJsonFormat[School] = jsonFormat1(School.apply)
}
case class ClassInfo
(
id: String,
classDetails: ClassDetails,
school: School
)
object ClassInfo {
implicit object ClassInfoFormat extends RootJsonFormat[ClassInfo] {
def write(c: ClassInfo): JsValue = JsObject(
"id" -> JsString(c.id),
"classDetails" -> c.classDetails.toJson,
"school" -> c.school.toJson
)
def read(value: JsValue): ClassInfo = {
value.asJsObject.getFields("id", "classDetails", "school") match {
case Seq(JsString(name), details, school) =>
new ClassInfo(name, details.convertTo[ClassDetails], school.convertTo[School])
case _ => throw new DeserializationException("ClassInfo expected")
}
}
}
}
val json = """[{"id":"my-id","classDetails":{"sec":"2","teacher":"John"},"school":{"name":"newschool"}}]"""
// JSON string to case classes
val classInfos = json.parseJson.convertTo[Seq[ClassInfo]]
classInfos.zipWithIndex.foreach { case (c, idx) =>
println(s"$idx => $c")
}
println
// Seq[ClassInfo] to JSON
println(s"$classInfos: ")
println(classInfos.toJson.prettyPrint)
I am attempting to migrate a Rails/Mongodb application to Play 2.3 using play-reactivemongo and reactivemongo-extensions. In modeling my data I am running across a problem serializing and deserializing a Map[Int,Boolean].
When I try to define my formats via macro like so
implicit val myCaseClass = Json.format[MyCaseClass]
where MyCaseClass has a few string fields, a BSONObjectID field, and a Map[Int,Boolean] field the compiler complains with:
No Json serializer found for type Map[Int,Boolean]. Try to implement an implicit Writes or Format for this type.
No Json deserializer found for type Map[Int,Boolean]. Try to implement an implicit Reads or Format for this type.
Looking at the source code for Play in Reads.scala I see a Reads defined for Map[String,_] but none for Map[Int,_].
Is there a reason why Play has default Read/Writes for string maps but not for other simple types?
I don't fully understand the Map[String,_] defined by play because I am fairly new to scala. How would I go about translating that into a Map[Int,_]? If that is not possible for some technical reason how would I define a Reads/Writes for Map[Int,Boolean]?
you can write your own reads and writes in play.
in your case, this would look like this:
implicit val mapReads: Reads[Map[Int, Boolean]] = new Reads[Map[Int, Boolean]] {
def reads(jv: JsValue): JsResult[Map[Int, Boolean]] =
JsSuccess(jv.as[Map[String, Boolean]].map{case (k, v) =>
Integer.parseInt(k) -> v .asInstanceOf[Boolean]
})
}
implicit val mapWrites: Writes[Map[Int, Boolean]] = new Writes[Map[Int, Boolean]] {
def writes(map: Map[Int, Boolean]): JsValue =
Json.obj(map.map{case (s, o) =>
val ret: (String, JsValueWrapper) = s.toString -> JsBoolean(o)
ret
}.toSeq:_*)
}
implicit val mapFormat: Format[Map[Int, Boolean]] = Format(mapReads, mapWrites)
I have tested it with play 2.3. I'm not sure if it's the best approach to have a Map[Int, Boolean] on server side and a json object with string -> boolean mapping on the client side, though.
JSON only allows string keys (a limitation it inherits from JavaScript).
Play Json provides built-in mapReads and mapWrites for reading and writing Maps.
mapReads takes a (String => JsResult[K]) to let you convert the key to your custom type.
mapWrites returns a Writes[Map[String, Boolean]], and you can use contramap to modify that writer into one that works with a Map[Int, Boolean]
import play.api.libs.json.{JsResult, Reads, Writes}
import scala.util.Try
import play.api.libs.json.Reads.mapReads
import play.api.libs.json.MapWrites.mapWrites
object MapExample {
implicit val reads: Reads[Map[Int, Boolean]] =
mapReads[Int, Boolean](s => JsResult.fromTry(Try(s.toInt)))
implicit val writes: Writes[Map[Int, Boolean]] =
mapWrites[Boolean].contramap(_.map { case (k, v) => k.toString -> v})
}
Thanks to Seth Tisue. This is my "generics" (half) way.
"half" because it does not handle a generic key. one can copy paste and replace the "Long" with "Int"
"Summary" is a type I've wanted to serialize (and it needed its own
serializer)
/** this is how to create reader and writer or format for Maps*/
// implicit val mapReads: Reads[Map[Long, Summary]] = new MapLongReads[Summary]
// implicit val mapWrites: Writes[Map[Long, Summary]] = new MapLongWrites[Summary]
implicit val mapLongSummaryFormat: Format[Map[Long, Summary]] = new MapLongFormats[Summary]
This is the required implementation:
class MapLongReads[T]()(implicit reads: Reads[T]) extends Reads[Map[Long, T]] {
def reads(jv: JsValue): JsResult[Map[Long, T]] =
JsSuccess(jv.as[Map[String, T]].map{case (k, v) =>
k.toString.toLong -> v .asInstanceOf[T]
})
}
class MapLongWrites[T]()(implicit writes: Writes[T]) extends Writes[Map[Long, T]] {
def writes(map: Map[Long, T]): JsValue =
Json.obj(map.map{case (s, o) =>
val ret: (String, JsValueWrapper) = s.toString -> Json.toJson(o)
ret
}.toSeq:_*)
}
class MapLongFormats[T]()(implicit format: Format[T]) extends Format[Map[Long, T]]{
override def reads(json: JsValue): JsResult[Map[Long, T]] = new MapLongReads[T].reads(json)
override def writes(o: Map[Long, T]): JsValue = new MapLongWrites[T].writes(o)
}
We can generalize the solution of 3x14159265 and Seth Tisue thanks to 2 small type classes:
import play.api.libs.json.Json.JsValueWrapper
import play.api.libs.json._
import simulacrum._
object MapFormat {
#typeclass trait ToString[A] {
def toStringValue(v: A): String
}
#typeclass trait FromString[A] {
def fromString(v: String): A
}
implicit final def mapReads[K: FromString, V: Reads]: Reads[Map[K, V]] =
new Reads[Map[K, V]] {
def reads(js: JsValue): JsResult[Map[K, V]] =
JsSuccess(js.as[Map[String, V]].map { case (k, v) => FromString[K].fromString(k) -> v })
}
implicit final def mapWrites[K: ToString, V: Writes]: Writes[Map[K, V]] =
new Writes[Map[K, V]] {
def writes(map: Map[K, V]): JsValue =
Json.obj(map.map {
case (s, o) =>
val ret: (String, JsValueWrapper) = ToString[K].toStringValue(s) -> o
ret
}.toSeq: _*)
}
implicit final def mapFormat[K: ToString: FromString, V: Format]: Format[Map[K, V]] = Format(mapReads, mapWrites)
}
Note that I use Simulacrum (https://github.com/mpilquist/simulacrum) to define my type classes.
Here is an example of how to use it:
final case class UserId(value: String) extends AnyVal
object UserId {
import MapFormat._
implicit final val userToString: ToString[UserId] =
new ToString[UserId] {
def toStringValue(v: UserId): String = v.value
}
implicit final val userFromString: FromString[UserId] =
new FromString[UserId] {
def fromString(v: String): UserId = UserId(v)
}
}
object MyApp extends App {
import MapFormat._
val myMap: Map[UserId, Something] = Map(...)
Json.toJson(myMap)
}
if IntelliJ says that your import MapFormat._ is never used, you can and this: implicitly[Format[Map[UserId, Something]]] just below the import. It'll fix the pb. ;)
A specific KeyWrites and KeyReads is available in play-json 2.9.x
private implicit val longKeyWrites = KeyWrites[Int](_.toString)
private implicit val longKeyReads =
KeyReads[Int](str => Try(str.toInt).fold(e => JsError(e.getMessage), JsSuccess(_)))
Json.obj("1" -> "test").validate[Map[Int,String]] // JsSuccess(Map(1 -> test))
Like the accepted answer - a bit shorter:
implicit val mapReads: Reads[Map[Int, Boolean]] = (jv: JsValue) =>
JsSuccess(jv.as[Map[String, Boolean]].map { case (k, v) =>
k.toInt -> v
})
implicit val mapWrites: Writes[Map[Int, Boolean]] = (map: Map[Int, Boolean]) =>
Json.toJson(map.map { case (s, o) =>
s.toString -> o
})
implicit val jsonMapFormat: Format[Map[Int, Boolean]] = Format(mapReads, mapWrites)
Here a little test:
val json = Json.toJson(Map(1 -> true, 2 -> false))
println(json) // {"1":true,"2":false}
println(json.validate[Map[Int, Boolean]]) // JsSuccess(Map(1 -> true, 2 -> false),)
https://gist.github.com/fancellu/0bea53f1a1dda712e179892785572ce3
Here is a way to persist a Map[NotString,...]
I have a case class Foo(bars: List[Bar]) who is rendered as json via Json inception as an object with an array :
{"bars": [
{
"key: "4587-der",
"value": "something"
}
]
}
But I want to render the bars: List[Bar] as a "map" where Bar.key is used as key :
{"bars":{
"4587-der": {
"value": "something"
}
}
}
How can I obtains that without modifying my case class Foo ?
Thanks a lot
You can define a Writes for Bar by extending Writes[Bar] and implementing a writes method for it:
case class Bar(key: String, value: String)
implicit object BarWrites extends Writes[Bar] {
def writes(bar: Bar): JsValue = Json.obj(
bar.key -> Json.obj("value" -> bar.value)
)
}
scala> Json.stringify(Json.toJson(Bar("4587-der", "something")))
res0: String = {"4587-der":{"value":"something"}}
For those that may be interested, here is a (somewhat) crude implementation of Reads[Bar]:
implicit object BarReads extends Reads[Bar] {
def reads(js: JsValue): JsResult[Bar] = js match {
case JsObject(Seq((key, JsObject(Seq(("value", JsString(value))))))) => JsSuccess(Bar(key, value))
case _ => JsError(Seq())
}
}
scala> Json.parse(""" [{"4587-der":{"value": "something"}}] """).validate[List[Bar]]
res11: play.api.libs.json.JsResult[List[Bar]] = JsSuccess(List(Bar(4587-der,something)),)
Edit, since the OP wants the Bars merged into an object rather than an array:
You'll also have to define a special Writes[List[Bar]] as well:
implicit object BarListWrites extends Writes[List[Bar]] {
def writes(bars: List[Bar]): JsValue =
bars.map(Json.toJson(_).as[JsObject]).foldLeft(JsObject(Nil))(_ ++ _)
}
scala> val list = List(Bar("4587-der", "something"), Bar("1234-abc", "another"))
list: List[Bar] = List(Bar(4587-der,something), Bar(1234-abc,another))
scala> Json.stringify(Json.toJson(list))
res1: String = {"4587-der":{"value":"something"},"1234-abc":{"value":"another"}}
I'm writing a simple scala application that opens a flat file of json data, parses it and finally prints it out to the screen.
The next step will require that I stop at each object and add another item (string) to the front of it. My question is how can I add a new string per object in this list?
The following is my JSON implementation (credit goes to the init author here)
import scala.util.parsing.combinator._
class JSON extends JavaTokenParsers {
def obj: Parser[Map[String, Any]] =
"{"~> repsep(member, ",") <~"}" ^^ (Map() ++ _)
def arr: Parser[List[Any]] =
"["~> repsep(value, ",") <~"]"
def member: Parser[(String, Any)] =
stringLiteral~":"~value ^^
{ case name~":"~value => (name, value) }
def value: Parser[Any] = (
obj
| arr
| stringLiteral
| floatingPointNumber ^^ (_.toInt)
| "null" ^^ (x => null)
| "true" ^^ (x => true)
| "false" ^^ (x => false)
)
}
Next I call this w/ a flat file like so
import java.io.FileReader
import scala23.JSON
class JSONTest extends JSON {
def main(args: String) {
val reader = new FileReader(args)
println(parseAll(value, reader))
}
}
Then I get a valid println of the json contents. Instead I would like to pass this parse method a String and have it append it or create a new json object that has the string at the front of each object inside
Update
My current attempt looks something like the below
class JSONTest extends JSON {
def main(args: String) {
val reader = new FileReader(args)
val header = ("abc", "def")
// println(parseAll(value, reader).map(addHeader(_, header)))
println(parseAll(value, reader).map(addHeader(_.asInstanceOf[Map[String, Any]], header)))
}
def addHeader(xyz:Map[String, Any], header:(String, Any)):Map[String, Any] = {
xyz.map {
case (k, m:Map[String, Any]) => (k, addHeader(m))
case e => e
} + header
}
}
But I'm currently getting a few errors in Intellij
error: missing parameter type for expanded function ((x$1) => x$1.asInstanceOf[Map[String, Any]])
println(parseAll(value, reader).map(addHeader(_.asInstanceOf[Map[String, Any]], header)))
AND
error: not enough arguments for method addHeader: (xyz: Map[String,Any],header: (String, Any))Map[String,Any].
Unspecified value parameter header.
case (k, m:Map[String, Any]) => (k, addHeader(m))
Any help would be much appreciated (thank you in advance!)
Have you tried using map on the parser output instead.
Edit: this compiles on my machine
import java.io.FileReader
import scala23.JSON
class JSONTest extends JSON {
def main(args: String) {
val reader = new FileReader(args)
val header = ("abc", "def")
// println(parseAll(value, reader).map(addHeader(_, header)))
println(parseAll(value, reader).map(addHeader(_, header)))
}
def addHeader(xyz:Any, header:(String, Any)):Any = xyz match {
case obj:Map[String, Any] => obj.map {
case (k, m:Map[String, Any]) => (k, addHeader(m, header))
case e => e
} + header
case arr:List[Any] => arr.map(addHeader(_, header))
case e => e
}
}
It should be handling the varied output of the parse better.