I want to upload a JSON array of entities using Play.
My model looks like this:
case class Pet(name: String, age: Int)
object Pet {
implicit val petReads: Reads[Pet] = (
(JsPath \ "name").read[String](minLength[String](2)) and
(JsPath \ "age").read[Int](min(0))
)(Pet.apply _)
)
My JSON input is a JSON array of entries. For example:
'[{"name": "Scooby","age":7},{"name": "Toothless","age": 3}]'
The action for working on the entries is this:
def create = Action(BodyParsers.parse.json) { implicit request =>
val entries = request.body.validate[Seq[Pet]]
entries.fold(errors => {BadRequest(Json.obj("status" -> "Bad Request", "message" -> JsError.toJson(errors)))},
elements => {//do something with it
Ok(Json.obj("status" -> "OK", "message" -> (Json.toJson("Done."))))})
}
I want my validation to be able to detect value problems. For example, if the string.length < 2 or the age number is negative.
However it doesn't work for arrays with .validate[Seq[Pet]]. Entries with a name of length < 2 can get past the validation.
If I try to upload each entry individually as a simple JSON entry (not a json array) and use .validate[Pet] instead, everything works fine. Any tips on how to tweak the validation so that it works for arrays?
Found the solution, just use .validate[Array[Pet]] and it works out of the box.
Related
I have many very large json-objects that I return from Play Framework with Scala.
In most cases the user doesn't need all the data in the objects, only a few fields. So I want to pass in the paths I need (as query parameters), and return a subset of the json object.
I have looked at using JSON Transformers for this task.
Filter code
def filterByPaths(paths: List[JsPath], inputObject: JsObject) : JsObject = {
paths
.map(_.json.pick)
.map(inputObject.transform)
.filter(_.isSuccess)
.map { case JsSuccess(value, path) => (value, path) }
.foldLeft(Json.obj()) { (obj, jsValueAndPath) =>
val(jsValue, path) = jsValueAndPath
val transformer = __.json.update(path.json.put(jsValue))
obj.transform(transformer).get
}
}
Usage:
val input = Json.obj(
"field1" -> Json.obj(
"field2" -> "right result"
),
"field4" -> Json.obj(
"field5" -> "not included"
),
)
val result = filterByPaths(List(JsPath \ "field1" \ "field2"), input)
// {"field1":{"field2":"right result"}}
Problem
This code works fine for JsObjects. But I can't make it work if there are JsArrays in the strucure. I had hoped that my JsPath could contain an index to look up the field, but that's not the case. (Don't know why I assumed that, maybe my head was too far in the JavaScript-world)
So this would fail to return the first entry in the Array:
val input: JsObject = Json.parse("""
{
"arr1" : [{
"field1" : "value1"
}]
}
""").as[JsObject]
val result = filterByPaths(List(JsPath \ "arr1" \ "0"), input)
// {}
Question
My question is: How can I return a subset of a json structure that contains arrays?
Alternative solution
I have the data as a case class first, and I serialize it to Json, and then run filterByPaths on it. Having a Reader that only creates the json I need in the first place might be a better solution, but creating a Reader on the fly, with configuration from queryparams seamed a more difficult task, then just stripping down the json afterwards.
The example of the returning array element:
val input: JsValue = Json.parse("""
{
"arr1" : [{
"field1" : "value1"
}]
}
""")
val firstElement = (input \ "arr1" \ 0).get
val firstElementAnotherWay = input("arr1")(0)
More about this in the Play Framework documentation: https://www.playframework.com/documentation/2.6.x/ScalaJson
Update
It looks like you got the old issue RuntimeException: expected KeyPathNode. JsPath.json.put, JsPath.json.update can't past an object to a nesting array.
https://github.com/playframework/playframework/issues/943
https://github.com/playframework/play-json/issues/82
What you can do:
Use the JSZipper: https://github.com/mandubian/play-json-zipper
Create a script to update arrays "manually"
If you can afford it, strip array in a resulting object
Example of stripping array (point 3):
def filterByPaths(paths: List[JsPath], inputObject: JsObject) : JsObject = {
paths
.map(_.json.pick)
.map(inputObject.transform)
.filter(_.isSuccess)
.map { case JsSuccess(value, path) => (value, path)}
.foldLeft(Json.obj()) { (obj, jsValueAndPath) =>
val (jsValue, path) = jsValueAndPath
val arrayStrippedPath = JsPath(path.path.filter(n => !(n.toJsonString matches """\[\d+\]""")))
val transformer = __.json.update(arrayStrippedPath.json.put(jsValue))
obj.transform(transformer).get
}
}
val result = filterByPaths(List(JsPath \ "arr1" \ "0"), input)
// {"arr1":{"field1":"value1"}}
The example
The best to handle JSON objects is by using case classes and create implicit Reads and Writes, by that you can handle errors every fields directly. Don't make it complicated.
Don't use .get() much recommended to use .getOrElse() because scala is a type-safe programming language.
Don't just use any Libraries except you know the process behind it, much better to create your own parsing method with simplified solution to save memory.
I hope it will help you..
Hi everyone recently I faced an issue in converting json into my own data model.
I have a json format message which may contain an empty string:
{
"name" : "John Doe",
"hobbies": ""
}
or a list of hobby types:
{
"name" : "John Doe",
"hobbies": [{"name":"basketball"}]
}
And the following is my case class data model in scala play framework:
case class Person(name: String, hobbies: List[Hobby])
case class Hobby(name: String)
Right now I'm using the default json formatter but of course it's not working well when we have empty string as value.
implicit val HobbyJson= Json.format[Hobby]
implicit val PersonJson = Json.format[Person]
it will throw exception if the hobbies has a empty string. I want to convert it into an empty list when it's the empty string. I search the document Play provides but couldn't find infomation. Can anyone give some suggestions?
Thanks in advance.
As you mentioned, the default Format macros won't work for you here because of the inconsistent treatment of hobbies. So you need to implement your own Reads[Person] - here's how I'd do it:
object PersonJson {
implicit val hobbyConverter = Json.format[Hobby]
val personReads = new Reads[Person] {
override def reads(json: JsValue): JsResult[Person] = {
for {
personName <- (json \ "name").validate[String]
hobbies <- (json \ "hobbies").validate[JsValue]
} yield {
val maybeHobbyList = hobbies.validate[List[Hobby]].asOpt
Person(personName, maybeHobbyList.getOrElse(Nil))
}
}
}
implicit val personConverter = Format(personReads, Json.writes[Person])
}
The key thing to note here is surrounding the whole thing in a JsResult courtesy of the for-comprehension and the yield. This gives us all the necessary checking (like the name field being there and being a String, and the hobbies field being there).
The code within the yield block only runs if we've got something that looks pretty close to a Person. Then we can safely try validating the hobbies as a List[Hobby], and convert the result to an Option[List[Hobby]]. It'll be a None if it didn't work (thus it must have been a string) and so we default it to the empty list as required.
Thanks #millhouse answer, it definitely works. Like he said we need a custom Reads[Person] to properly convert it.
I also post my code as reference.
implicit val personJsonReads: Reads[Person] = (
(__ \ "name").read[String] and
(__ \ "hobbies").read[List[Hobby]].orElse(Reads.pure(List()))
) (Person.apply _)
read[List[Hobby]].orElse(Reads.pure(List())) will generate the empty list when the value cannot convert to List[Hobby].
I've been asked to parse a JSON file to get all the buses that are over a specified speed inputed by the user.
The JSON file can be downloaded here
It's like this:
{
"COLUMNS": [
"DATAHORA",
"ORDEM",
"LINHA",
"LATITUDE",
"LONGITUDE",
"VELOCIDADE"
],
"DATA": [
[
"04-16-2015 00:00:55",
"B63099",
"",
-22.7931,
-43.2943,
0
],
[
"04-16-2015 00:01:02",
"C44503",
781,
-22.853649,
-43.37616,
25
],
[
"04-16-2015 00:11:40",
"B63067",
"",
-22.7925,
-43.2945,
0
],
]
}
The thing is: I'm really new to scala and I have never worked with json before (shame on me). What I need is to get the "Ordem", "Linha" and "Velocidade" from DATA node.
I created a case class to enclousure all the data so as to later look for those who are over the specified speed.
case class Bus(ordem: String, linha: Int, velocidade: Int)
I did this reading the file as a textFile and spliting. Although this way, I need to foreknow the content of the file in order to go to the lines after DATA node.
I want to know how to do this using a JSON parser. I've tried many solutions, but I couldn't adapt to my problem, because I need to extract all the lines from DATA node instead of nodes inside one node.
Can anyone help me?
PS: Sorry for my english, not a native speaker.
First of all, you need to understand the different JSON data types. The basic types in JSON are numbers, strings, booleans, arrays, and objects. The data returned in your example is an object with two keys: COLUMNS and DATA. The COLUMNS key has a value that is an array of strings and numbers. The DATA key has a value which is an array of arrays of strings.
You can use a library like PlayJSON to work with this type of data:
val js = Json.parse(x).as[JsObject]
val keys = (js \ "COLUMNS").as[List[String]]
val values = (js \ "DATA").as[List[List[JsValue]]]
val busses = values.map(valueList => {
val keyValues = (keys zip valueList).toMap
for {
ordem <- keyValues("ORDEM").asOpt[String]
linha <- keyValues("LINHA").asOpt[Int]
velocidade <- keyValues("VELOCIDADE").asOpt[Int]
} yield Bus(ordem, linha, velocidade)
})
Note the use of asOpt when converting the properties to the expected types. This operator converts the key-values to the provided type if possible (wrapped in Some), and returns None otherwise. So, if you want to provide a default value instead of ignoring other results, you could use keyValues("LINHA").asOpt[Int].getOrElse(0), for example.
You can read more about the Play JSON methods used here, like \ and as, and asOpt in their docs.
You can use Spark SQL to achieve it. Refer section under JSON Datasets here
In essence, Use spark APIs to load a JSON and register it as temp table.
You can run your SQL queries on the table from there.
As seen on #Ben Reich answer, that code works great. Thank you very much.
Although, my Json had some type problems on "Linha". As it can be seen on the JSON example that I put on the Question, there are "" and also numbers, e.g., 781.
When trying to do keyValues("LINHA").asOpt[Int].getOrElse(0), it was producing an error saying that value flatMap is not a member of Int.
So, I had to change some things:
case class BusContainer(ordem: String, linha: String, velocidade: Int)
val jsonString = fromFile("./project/rj_onibus_gps.json").getLines.mkString
val js = Json.parse(jsonString).as[JsObject]
val keys = (js \ "COLUMNS").as[List[String]]
val values = (js \ "DATA").as[List[List[JsValue]]]
val buses = values.map(valueList => {
val keyValues = (keys zip valueList).toMap
println(keyValues("ORDEM"),keyValues("LINHA"),keyValues("VELOCIDADE"))
for {
ordem <- keyValues("ORDEM").asOpt[String]
linha <- keyValues("LINHA").asOpt[Int].orElse(keyValues("LINHA").asOpt[String])
velocidade <- keyValues("VELOCIDADE").asOpt[Int]
} yield BusContainer(ordem, linha.toString, velocidade)
})
Thanks for the help!
I am trying to send data from the client to the server using a JSON request. The body of the JSON request looks like this:
[
[
{"x":"0","y":"0","player":0},
{"x":"0","y":"1","player":0},
{"x":"0","y":"2","player":1}
],
[
{"x":"1","y":"0","player":0},
{"x":"1","y":"1","player":2},
{"x":"1","y":"2","player":0}
],
[
{"x":"2","y":"0","player":0},
{"x":"2","y":"1","player":1},
{"x":"2","y":"2","player":2}
]
]
On server side I would like to transform data with Play 2 framework to Scala 2D list like this:
List(
List(0,0,1),
List(0,2,0),
List(0,1,2)
)
this is 3x3 but it can be variable like 50x50 or so.
Thanks for any help.
It might be incomplete (don't know if you want to modelize the square matrix contraint as well) but something like that could be a good start:
First here is what the controller (and model) part can define
import play.api.libs.json.Json._
import play.api.libs.json._
type PlayerT = (String, String, Int)
implicit val playerTripleReads:Reads[PlayerT] = (
(__ \ "x").read[String] and
(__ \ "y").read[String] and
(__ \ "player").read[Int]
tupled
)
def getJson = Action(parse.json) { request =>
request.body.validate[List[List[PlayerT]]].map{
case xs => Ok(xs.mkString("\n"))
}.recoverTotal{
e => BadRequest("Detected error:"+ JsError.toFlatJson(e))
}
}
In this version, you'll get a list of list holding validated tuples of the form (String, String, Int) which has been aliased with the PlayerT type to save some typing.
As you may saw, the reader as been created "by-hand" by composing (using the and combinator) three basic blocks and the result is flattened using the tupled operator.
With this solution you're now on track to play with those tuples, but IMO the code will suffer from bad readability, because of the usage of _1, _2 and _3 along the way.
So here is a different approach (which is in fact even easier...) that tackles this problem of sane coding, this will simply defined a `case class that models your atomic data
case class Player(x:String, y:String, player:Int)
implicit val playerReads = Json.reads[Player]
def getJson = Action(parse.json) { request =>
request.body.validate[List[List[Player]]].map{
case xs => Ok(xs.mkString("\n"))
}.recoverTotal{
e => BadRequest("Detected error:"+ JsError.toFlatJson(e))
}
}
Note that, the reader will always follow further changes in your data representation, that is the case class's fields thanks to the use of the implicit creation of the reader at compile time.
Now, you'll be able to use x, y and player fields rather than _1, _2 and _3.
Here is a JSON list I want to process:
scala> jsonStructure \ "response" \ "docs"
res4: play.api.libs.json.JsValue = [{"title":"the very first document"},{"title":"on brick walls"}]
I tried converting it to a list, but I got something with different semantics, a list whose only element is that list:
scala> jsonStructure \ "response" \\ "docs"
res3: Seq[play.api.libs.json.JsValue] = List([{"title":"the very first document"},{"title":"on brick walls"}])
scala> res3.size
res4: Int = 1
I tried this kludge, which does the trick:
scala> (jsonStructure \ "response" \ "docs").as[Seq[play.api.libs.json.JsValue]]
res9: Seq[play.api.libs.json.JsValue] = List({"title":"the very first document"}, {"title":"on brick walls"})
scala> res9 size
res10: Int = 2
Why did the \\ not work? What is the idiomatic way to understand a JsValue into a JsArray? While still maintaining the "navigating using and \ never fails" philosophy? I want to parse deeper structures, like a list of obj inside an obj which was a list element; I want a method that wont become unwieldy for deeply nested structures.
Feel free to correct my approach if you find it complicated, brittle etc.
If you wanted just the list of title entries, you probably would want to write something like:
json \\ "title"
Which returns:
List("the very first document", "on brick walls")
The \\ works by listing any element matching your path selector (at current level and all the descendants). Since there is essentially one docs element, it returns a list with one element. Only when you ask for a title, it will return a list of titles.
But from you approach you probably wanted to pattern-match a JsValue on JsArray:
def convert(json : JsValue) : Option[Seq[JsValue]] = json match {
case JsArray(arr) => Some(arr)
case _ => None
}
It returns exactly what you wanted:
List({"title":"the very first document"}, {"title":"on brick walls"})