Handle multidimensional JSON with scala Play framework - json

I am trying to send data from the client to the server using a JSON request. The body of the JSON request looks like this:
[
[
{"x":"0","y":"0","player":0},
{"x":"0","y":"1","player":0},
{"x":"0","y":"2","player":1}
],
[
{"x":"1","y":"0","player":0},
{"x":"1","y":"1","player":2},
{"x":"1","y":"2","player":0}
],
[
{"x":"2","y":"0","player":0},
{"x":"2","y":"1","player":1},
{"x":"2","y":"2","player":2}
]
]
On server side I would like to transform data with Play 2 framework to Scala 2D list like this:
List(
List(0,0,1),
List(0,2,0),
List(0,1,2)
)
this is 3x3 but it can be variable like 50x50 or so.
Thanks for any help.

It might be incomplete (don't know if you want to modelize the square matrix contraint as well) but something like that could be a good start:
First here is what the controller (and model) part can define
import play.api.libs.json.Json._
import play.api.libs.json._
type PlayerT = (String, String, Int)
implicit val playerTripleReads:Reads[PlayerT] = (
(__ \ "x").read[String] and
(__ \ "y").read[String] and
(__ \ "player").read[Int]
tupled
)
def getJson = Action(parse.json) { request =>
request.body.validate[List[List[PlayerT]]].map{
case xs => Ok(xs.mkString("\n"))
}.recoverTotal{
e => BadRequest("Detected error:"+ JsError.toFlatJson(e))
}
}
In this version, you'll get a list of list holding validated tuples of the form (String, String, Int) which has been aliased with the PlayerT type to save some typing.
As you may saw, the reader as been created "by-hand" by composing (using the and combinator) three basic blocks and the result is flattened using the tupled operator.
With this solution you're now on track to play with those tuples, but IMO the code will suffer from bad readability, because of the usage of _1, _2 and _3 along the way.
So here is a different approach (which is in fact even easier...) that tackles this problem of sane coding, this will simply defined a `case class that models your atomic data
case class Player(x:String, y:String, player:Int)
implicit val playerReads = Json.reads[Player]
def getJson = Action(parse.json) { request =>
request.body.validate[List[List[Player]]].map{
case xs => Ok(xs.mkString("\n"))
}.recoverTotal{
e => BadRequest("Detected error:"+ JsError.toFlatJson(e))
}
}
Note that, the reader will always follow further changes in your data representation, that is the case class's fields thanks to the use of the implicit creation of the reader at compile time.
Now, you'll be able to use x, y and player fields rather than _1, _2 and _3.

Related

How do I transform a JSON fields into a Seq in Scala Play framework 2?

I have some JSON coming from an external API which I have no control over. Part of the JSON is formatted like this:
{
"room_0": {
"area_sq_ft": 151.2
},
"room_1": {
"area_sq_ft": 200.0
}
}
Instead of using an array like they should have, they've used room_n for a key to n number of elements. Instead of creating a case class with room_0, room_1, room_2, etc., I want to convert this to a Seq[Room] where this is my Room case class:
case class Room(area: Double)
I am using Reads from play.api.libs.json for converting other parts of the JSON to case classes and would prefer to use Reads for this conversion. How could I accomplish that?
Here's what I've tried.
val sqFtReads = (__ \ "size_sq_ft").read[Double]
val roomReads = (__ \ "size_sq_ft").read[Seq[Room]](sqFtReads).map(Room)
cmd19.sc:1: overloaded method value read with alternatives:
(t: Seq[$sess.cmd17.Room])play.api.libs.json.Reads[Seq[$sess.cmd17.Room]] <and>
(implicit r: play.api.libs.json.Reads[Seq[$sess.cmd17.Room]])play.api.libs.json.Reads[Seq[$sess.cmd17.Room]]
cannot be applied to (play.api.libs.json.Reads[Double])
val roomReads = (__ \ "size_sq_ft").read[Seq[Room]](sqFtReads).map(Room)
A tricky little challenge but completely achievable with Reads.
First, Reads[Room] - i.e. the converter for a single Room instance:
val roomReads = new Reads[Room] {
override def reads(json: JsValue): JsResult[Room] = {
(json \ "area_sq_ft").validate[Double].map(Room(_))
}
}
Pretty straightforward; we peek into the JSON and try to find a top-level field called area_sq_ft which validates as a Double. If it's all good, we return the populated Room instance as needed.
Next up, the converter for your upstream object that in good Postel's Law fashion, you are cleaning up for your own consumers.
val strangeObjectReads = new Reads[Seq[Room]] {
override def reads(json: JsValue): JsResult[Seq[Room]] = {
json.validate[JsObject].map { jso =>
val roomsSortedNumerically = jso.fields.sortBy { case (name, contents) =>
val numericPartOfRoomName = name.dropWhile(!_.isDigit)
numericPartOfRoomName.toInt
}
roomsSortedNumerically.map { case (name, contents) =>
contents.as[Room](roomReads)
}
}
}
}
The key thing here is the json.validate[JsObject] around the whole lot. By mapping over this we get the JsResult that we need to wrap the whole thing, plus, we can get access to the fields inside the JSON object, which is defined as a Seq[(String, JsValue)].
To ensure we put the fields in the correct order in the output sequence, we do a little bit of string manipulation, getting the numeric part of the room_1 string, and using that as the sortBy criteria. I'm being a bit naive here and assuming your upstream server won't do anything nasty like skip room numbers!
Once you've got the rooms sorted numerically, we can just map over them, converting each one with our roomReads converter.
You've probably noticed that my custom Reads implementations are most definitely not one-liners. This comes from bitter experience dealing with oddball upstream JSON formats. Being a bit verbose, using a few more variables and breaking things up a bit pays off big time when that upstream server changes its JSON format suddenly!

Play Framework 2 cannot properly validate Json Arrays (code inside)

I want to upload a JSON array of entities using Play.
My model looks like this:
case class Pet(name: String, age: Int)
object Pet {
implicit val petReads: Reads[Pet] = (
(JsPath \ "name").read[String](minLength[String](2)) and
(JsPath \ "age").read[Int](min(0))
)(Pet.apply _)
)
My JSON input is a JSON array of entries. For example:
'[{"name": "Scooby","age":7},{"name": "Toothless","age": 3}]'
The action for working on the entries is this:
def create = Action(BodyParsers.parse.json) { implicit request =>
val entries = request.body.validate[Seq[Pet]]
entries.fold(errors => {BadRequest(Json.obj("status" -> "Bad Request", "message" -> JsError.toJson(errors)))},
elements => {//do something with it
Ok(Json.obj("status" -> "OK", "message" -> (Json.toJson("Done."))))})
}
I want my validation to be able to detect value problems. For example, if the string.length < 2 or the age number is negative.
However it doesn't work for arrays with .validate[Seq[Pet]]. Entries with a name of length < 2 can get past the validation.
If I try to upload each entry individually as a simple JSON entry (not a json array) and use .validate[Pet] instead, everything works fine. Any tips on how to tweak the validation so that it works for arrays?
Found the solution, just use .validate[Array[Pet]] and it works out of the box.

Abstraction to extract data from JSON in Scala

I am looking for a good abstraction to extract data form JSON (I am using json4s now).
Suppose I have a case class A and data in JSON format.
case class A(a1: String, a2: String, a3: String)
{"a1":"xxx", "a2": "yyy", "a3": "zzz"}
I need a function to extract the JSON data and return A with these data as follows:
val a: JValue => A = ...
I do not want to write the function a from scratch. I would rather compose it from primitive functions.
For example, I can write a primitive function to extract string by field name:
val str: (String, JValue) => String = {(fieldName, jval) => ... }
Now I would like to compose the function a: JValue => A from str. Does it make sense ?
Consider use of Play-JSON, which has a composable "Reads" object. If you've ever used ReactiveMongo, it can be used in much the same way. Contrary to some older posts here, it can be used stand-alone, without most of the rest of Play.
It uses the common "implicit translator" (my term) idiom. I found that my favorite deserializing pattern for using it is not highlighted in the docs, though - the pattern they espouse is a lot harder to get right, IMHO. I make heavy use of .as and .asOpt, which are documented on the first linked page above, in the small section "Using JsValue.as/asOpt". When deserializing a JSON object, you can say something like
val person:Person = (someParsedJsonObject \ "aPerson").as[Person]
and as long as you have an implicit Reads[Person] in scope, all just works. There are built-in Reads for all primitive types and many collection types. In many cases, it makes sense to put the Reads and Writes implicit objects in the companion object for, e.g., Person.
I thought json4s had a similar feature, but I could be wrong.
Argonaut is fully functional Scala library.
It allows to encode/decode case classes (JSON codecs).
import argonaut._, Argonaut._
case class Person(name: String, age: Int)
implicit def PersonDecodeJson: DecodeJson[Person]
jdecode2L(Person.apply)("name", "age")
// Codec for Person case class from JSON of form
// { "name": "string", "age": 1 }
It also provides JSON cursor (lenses/monocle) for custom parsing.
implicit def PersonDecodeJson: DecodeJson[Person] =
DecodeJson(c => for {
name <- (c --\ "_name").as[String]
age <- (c --\ "_age").as[String].map(_.toInt)
} yield Person(name, age))
// Decode Person from a JSON with property names different
// from those of the case class, and age passed as string:
// { "_name": "string", "age": "10" }
Parsing result is represented by DecodeResult type that can be composed (.map, .flatMap) and handle error cases.

Modifying JSON reads and writes in playframework 2.1

I'm a newbie and scala/play and need help with playframework's JSON reads/writes.
I use Json.reads[T] and Json.writes[T] macros to define json reads and writes for my classes. However I'd like to have one property name to be (always) mapped differently. Namely, I have property named id in my classes and I want it to be represented as _id when object is converted to json and vice versa.
Is there a way to modify reads/writes objects generated by Json.reads and Json.writes macros to achieve this or do I have to rewrite reads and writes manually just to have one property named differently?
EDIT
Let me try to explain the problem better. Consider the model object User:
case class User (id: BigInt, email: String, name: String)
When serializing User to json for purposes of serving json in context of a REST api the json should look like this:
{
"id": 23432,
"name": "Joe",
"email: "joe#example.com"
}
When serializing User to json for purposes of storing/updating/reading form MongoDB json should look like:
{
"_id": 23432,
"name": "Joe",
"email: "joe#example.com"
}
In other words everything is the same except when communicating with Mongo id should be represented as _id.
I know I could manually write two sets of reads and writes for each model object (one to be used for web and another for communication with Mongo) as suggested by Darcy Qiu in the answer, however maintaining two sets of reads and writes that are nearly identical except for the id property seems like a lot of code duplication so I'm wondering if there is a better approach.
First you define transformations for renames id/_id back and forth:
import play.api.libs.json._
import play.modules.reactivemongo.json._
val into: Reads[JsObject] = __.json.update( // copies the full JSON
(__ \ 'id).json.copyFrom( (__ \ '_id).json.pick ) // adds id
) andThen (__ \ '_id).json.prune // and after removes _id
val from: Reads[JsObject] = __.json.update( // copies the full JSON
(__ \ '_id).json.copyFrom( (__ \ 'id).json.pick ) // adds _id
) andThen (__ \ 'id).json.prune // and after removes id
(To understand why Reads is a transformation please read: https://www.playframework.com/documentation/2.4.x/ScalaJsonTransformers)
Assuming we have macro generated Writes and Reads for our entity class:
def entityReads: Reads[T] // eg Json.reads[Person]
def entityWrites: Writes[T] // eg Json.writes[Person]
Then we mix transformations with macro-generated code:
private[this] def readsWithMongoId: Reads[T] =
into.andThen(entityReads)
private[this] def writesWithMongoId: Writes[T] =
entityWrites.transform(jsValue => jsValue.transform(from).get)
The last thing. Mongo driver wants to be sure (ie typesafe-sure) that the json it inserts is a JsObject. That is why we need an OWrites. I haven't found better way than:
private[this] def oWritesWithMongoId = new OWrites[T] {
override def writes(o: T): JsObject = writesWithMongoId.writes(o) match {
case obj: JsObject => obj
case notObj: JsValue =>
throw new InternalError("MongoRepo has to be" +
"definded for entities which serialize to JsObject")
}
}
Last step is to privide an implicit OFormat.
implicit val routeFormat: OFormat[T] = OFormat(
readsWithMongoId,
oWritesWithMongoId
)
Let's say your case class, which is T in your question, is named User and has definision as below
case class User(_id: String, age: Int)
Your reads can be defined as
implicit val userReads = new Reads[User] {
def reads(js: JsValue): User = {
User(
(js \ "id").as[String],
(js \ "age").as[Int]
)
}
}
Your writes[User] should follow the same logic.
If you put in enough code you can achieve this with transformers:
val idToUnderscore = (JsPath).json.update((JsPath).read[JsObject].map { o:JsObject =>
o ++ o.transform((JsPath\"_id").json.put((JsPath\"id").asSingleJson(o))).get
}) andThen (JsPath\"id").json.prune
val underscoreWrite = normalWrites.transform( jsVal => jsVal.transform(idToUnderscore).get )
Here's the full test:
import play.api.libs.functional.syntax._
import play.api.libs.json._
val u = User("overlord", "Hansi Meier", "evil.overlord#hansi-meier.de")
val userToProperties = {u:User => (u.id, u.name, u.email)}
val normalWrites = (
(JsPath\"id").write[String] and
(JsPath\"name").write[String] and
(JsPath\"email").write[String]
)(userToProperties)
val idToUnderscore = (JsPath).json.update((JsPath).read[JsObject].map { o:JsObject =>
o ++ o.transform((JsPath\"_id").json.put((JsPath\"id").asSingleJson(o))).get
}) andThen (JsPath\"id").json.prune
val underscoreWrite = normalWrites.transform( jsVal => jsVal.transform(idToUnderscore).get )
info(Json.stringify(Json.toJson(u)(normalWrites)))
info(Json.stringify(Json.toJson(u)(underscoreWrite)))
Now, if you modify the normalWrites (say by adding additional properties), the underscoreWrite will still do what you want.

Suggestions for Writing Map as JSON file in Scala

I have a simple single key-valued Map(K,V) myDictionary that is populated by my program and at the end I want to write it as JSON format string in a text file - as I would need parse them later.
I was using this code earlier,
Some(new PrintWriter(outputDir+"/myDictionary.json")).foreach{p => p.write(compact(render(decompose(myDictionary)))); p.close}
I found it to be slower as the input size increased. Later, I used this var out = new
var out = new PrintWriter(outputDir+"/myDictionary.json");
out.println(scala.util.parsing.json.JSONObject(myDictionary.toMap).toString())
This is proving to be bit faster.
I have run this for sample input and found that this is faster than my earlier approach. I assuming my input map size would reach at least a million values( >1GB text file) (K,V) hence I want to make sure that I follow the faster and memory efficient approach for Map serialization process.What are other approaches that you would recommend,that I can look into to optimize this.
The JSON support in the standard Scala library is probably not the best choice. Unfortunately the situation with JSON libraries for Scala is a bit confusing, there are many alternatives (Lift JSON, Play JSON, Spray JSON, Twitter JSON, Argonaut, ...), basically one library for each day of the week... I suggest you have a look at these at least to see if any of them is easier to use and more performative.
Here is an example using Play JSON which I have chosen for particular reasons (being able to generate formats with macros):
object JsonTest extends App {
import play.api.libs.json._
type MyDict = Map[String, Int]
implicit object MyDictFormat extends Format[MyDict] {
def reads(json: JsValue): JsResult[MyDict] = json match {
case JsObject(fields) =>
val b = Map.newBuilder[String, Int]
fields.foreach {
case (k, JsNumber(v)) => b += k -> v.toInt
case other => return JsError(s"Not a (string, number) pair: $other")
}
JsSuccess(b.result())
case _ => JsError(s"Not an object: $json")
}
def writes(m: MyDict): JsValue = {
val fields: Seq[(String, JsValue)] = m.map {
case (k, v) => k -> JsNumber(v)
} (collection.breakOut)
JsObject(fields)
}
}
val m = Map("hallo" -> 12, "gallo" -> 34)
val serial = Json.toJson(m)
val text = Json.stringify(serial)
println(text)
val back = Json.fromJson[MyDict](serial)
assert(back == JsSuccess(m), s"Failed: $back")
}
While you can construct and deconstruct JsValues directly, the main idea is to use a Format[A] where A is the type of your data structure. This puts more emphasis on type safety than the standard Scala-Library JSON. It looks more verbose, but in end I think it's the better approach.
There are utility methods Json.toJson and Json.fromJson which look for an implicit format of the type you want.
On the other hand, it does construct everything in-memory and it does duplicate your data structure (because for each entry in your map you will have another tuple (String, JsValue)), so this isn't necessarily the most memory efficient solution, given that you are operating in the GB magnitude...
Jerkson is a Scala wrapper for the Java JSON library Jackson. The latter apparently has the feature to stream data. I found this project which says it adds streaming support. Play JSON in turn is based on Jerkson, so perhaps you can even figure out how to stream your object with that. See also this question.