The below code compiles, but throws an error: Exception in thread "main" scala.MatchError:[{"id":6430758,"name":...] (of class play.api.libs.json.JsArray). How can I read JSON for the given link by taking the items list in it and only 5 elements?
import play.api.libs.json._
def getProjects: List[Map[String, Any]] = {
val iter = getJSON("https://api.github.com/search/repositories?q=scala")
val json: JsValue = Json.parse(iter.get mkString "\n")
val projects = (json \ "items") match {
case l: List[Map[String, Any]] => l take 5
}
projects
}
def getJSON(url: String): Try[Iterator[String]] =
Try(Source.fromURL(url).getLines) recover {
case e: FileNotFoundException =>
throw new AppException(s"Requested page does not exist: ${e.getMessage}.")
case e: MalformedURLException =>
throw new AppException(s"Please make sure to enter a valid URL: ${e.getMessage}.")
case _ => throw new AppException("An unexpected error has occurred.")
}
Since you're using Play, you should work within its JsValue abstraction rather than jumping out to a Map[String, Any].
The reason your match is failing is because json \ "items" isn't a Map[String, Any], it's a JsValue. Ideally, you know the structure of your JSON (what your schema for project is) and you can deserialize to that:
case class Project(id: Long, name: String, ...)
object Project {
implicit val fmt = Json.format[Project]
}
val projects = WS.get("https://api.github.com/search/repositories?q=scala").map { response =>
response.json.validate[Map[String, Project]].map(_ take 5)
}
That leaves you with a Future[JsResult[Map[String, Project]]]. The outer type is Future because the operation is inherently asynchronous, JsResult will be either a JsSuccess with your Map[String, Project] or a JsError containing the reason(s) your JSON couldn't be validated.
It feels quick and dirty, but if that's really what you're wanting to do then you can try:
val listOfMaps: Seq[Map[String, String]] =
(res1 \ "items").as[JsArray].value.map { jsobj =>
jsobj.as[JsObject].value.map { case (key, value) =>
key -> value.toString
}
}.take(5)
A better option would be to create a case class with they keys and types that you are expecting and write a Reads to parse the Json to that case class. See https://www.playframework.com/documentation/2.3.x/ScalaJsonCombinators. Then you would have a list of your case class and you can easily take 5 from there.
Related
I have made a generic method which parses json to case class and it also works fine. But if tries to parse big json which have one or two mandatory field then I am not able to figure out which particular mandatory f ield is missing. I am only able to handle it with IllegalArgumentException. Is there a way to handle to know which is field is missing while parsing Json by using json4s.
Here is my code ->
object JsonHelper {
implicit val formats: DefaultFormats = DefaultFormats
def write[T <: AnyRef](value: T): String = jWrite(value)
def parse(value: String): JValue = jParser(value)
}
And this is the method I am using to parse Json and handle failed case ->
def parseJson[M](json: String)(implicit m: Manifest[M]): Either[ErrorResponse, M] = {
try
Right(JsonHelper.parse(json).extract[M])
catch {
case NonFatal(th) =>
th.getCause.getCause match {
case e: java.lang.IllegalArgumentException =>
error(s"Invalid JSON - $json", e)
Left(handle(exception = EmptyFieldException(e.getMessage.split(":").last)))
case _ =>
error(s"Invalid JSON - $json", th)
Left(handle(exception = new IllegalArgumentException("Invalid Json", th)))
}
}
}
Like for a Json ->
{
"name": "Json"
}
And case class ->
case class(name: String, profession: String)
if I try to parse above json into case class currently I am getting Invalid JSON - IllegalArgumentException. But is there a way that the exception tells which is field is missing like in above example "profession" is missing.
Is there a way to handle to know which is field is missing while parsing Json by using json4s.
Maybe you have more complicated setting, but for example for
import org.json4s._
import org.json4s.jackson.JsonMethods._
val str = """{
| "name": "Json"
|}""".stripMargin
val json = parse(str) // JObject(List((name,JString(Json))))
implicit val formats: Formats = DefaultFormats
case class MyClass(name: String, profession: String)
json.extract[MyClass]
it produces
org.json4s.MappingException: No usable value for profession
Did not find value which can be converted into java.lang.String
at org.json4s.reflect.package$.fail(package.scala:56)
at ...
with the name of missing field and if the class is just case class MyClass(name: String) then this produces MyClass(Json).
If the class is case class MyClass(name: String, profession: Option[String]) then this produces MyClass(Json,None).
So normally IllegalArgumentException should be followed by Caused by: org.json4s.MappingException with the field name. I guess now you're swallowing json4s MappingException somewhere. Maybe in th.getCause.getCause match .... It's hard to say without MCVE.
Let's say my API returns a JSON that looks like this:
{
"field1": "hey",
"field2": null,
}
I have this rule that only one of these fields can be null at the same time. In this example, only field2 is null so we're ok.
I can represent this in Scala with the following case class:
case class MyFields(
field1: Option[String],
field2: Option[String]
)
And by implementing some implicits and let circe do it's magic of converting the objects to JSON.
object MyFields {
implicit lazy val encoder: Encoder[MyFields] = deriveEncoder[MyFields]
implicit lazy val decoder: Decoder[MyFields] = deriveDecoder[MyFields]
Now, this strategy works. Kinda.
MyFields(Some("hey"), None)
MyFields(None, Some("hey"))
MyFields(Some("hey"), Some("hey"))
These all lead to JSONs that follow the rule. But it's also possible to do:
MyFields(None, None)
Which will lead to a JSON that breaks the rule.
So this strategy doesn't express the rule adequately. What's a better way to do it?
represent your data as having a member val field1or2 = Either[String, String]. If the circe built-in Codec.codecForEither doesn't meet your exact requirements, you could write your codec manually. From what you describe (field1 and fiel2 both must be present in json, one as string, one as null), something like
import io.circe.{Decoder, Encoder, HCursor, Json, DecodingFailure}
case class Fields(fields: Either[String, String])
implicit val encodeFields: Encoder[Fields] = new Encoder[Fields] {
final def apply(a: Fields): Json = Json.obj(a.fields match {
case Left(str) => {
"field1" -> Json.fromString(str)
"field2" -> Json.Null
}
case Right(str) => {
"field1" -> Json.Null
"field2" -> Json.fromString(str)
}
})
}
implicit val decodeFields: Decoder[Fields] = new Decoder[Fields] {
final def apply(c: HCursor): Decoder.Result[Fields] ={
val f1 = c.downField("field1").as[Option[String]]
val f2 = c.downField("field1").as[Option[String]]
(f1, f2) match {
case (Right(None), Right(Some(v2))) => Right(Fields(Right(v2)))
case (Right(Some(v1)), Right(None)) => Right(Fields(Left(v1)))
case (Left(failure), _) => Left(failure)
case (_, Left(failure)) => Left(failure)
case (Right(None), Right(None)) => Left(DecodingFailure("Either field1 or field2 must be non-null", Nil))
case (Right(Some(_)), Right(Some(_))) => Left(DecodingFailure("field1 and field2 may not both be non-null", Nil))
}
}
}
This is an example of application-level verification of data, along with range-checking values and other consistency checks. As such, it doesn't really belong with the raw JSON parsing, but needs to be done in a separate validation step.
So I recommend having a set of classes that directly represent the JSON data, and then a separate set of application classes that use application data types rather than JSON data types.
When reading data, the JSON is read into the data classes to check that the underlying JSON is valid. Then those data classes are converted to application classes with appropriate validation and conversion (checking value ranges, changing strings to enumerations etc.)
This allows the data format to be changed (e.g. to XML or database) without affecting the application logic.
(This is based on Martijn's answer and comment.)
Cats Ior datatype could be used, as following:
import cats.data.Ior
import io.circe.parser._
import io.circe.syntax._
import io.circe._
case class Fields(fields: Ior[String, String])
implicit val encodeFields: Encoder[Fields] = (a: Fields) =>
a.fields match {
case Ior.Both(v1, v2) => Json.obj(
("field1", Json.fromString(v1)),
("field2", Json.fromString(v2))
)
case Ior.Left(v) => Json.obj(
("field1", Json.fromString(v)),
("field2", Json.Null)
)
case Ior.Right(v) => Json.obj(
("field1", Json.Null),
("field2", Json.fromString(v))
)
}
implicit val decodeFields: Decoder[Fields] = (c: HCursor) => {
val f1 = c.downField("field1").as[Option[String]]
val f2 = c.downField("field2").as[Option[String]]
(f1, f2) match {
case (Right(Some(v1)), Right(Some(v2))) => Right(Fields(Ior.Both(v1, v2)))
case (Right(Some(v1)), Right(None)) => Right(Fields(Ior.Left(v1)))
case (Right(None), Right(Some(v2))) => Right(Fields(Ior.Right(v2)))
case (Left(failure), _) => Left(failure)
case (_, Left(failure)) => Left(failure)
case (Right(None), Right(None)) => Left(DecodingFailure("At least one of field1 or field2 must be non-null", Nil))
}
}
println(Fields(Ior.Right("right")).asJson)
println(Fields(Ior.Left("left")).asJson)
println(Fields(Ior.both("right", "left")).asJson)
println(parse("""{"field1": null, "field2": "right"}""").flatMap(_.as[Fields]))
I have a JsArray which contains JsValue objects representing two different types of entities - some of them represent nodes, the other part represents edges.
On the Scala side, there are already case classes named Node and Edge whose supertype is Element. The goal is to transform the JsArray (or Seq[JsValue]) to a collection that contains the Scala types, e.g. Seq[Element] (=> contains objects of type Node and Edge).
I have defined Read for the case classes:
implicit val nodeReads: Reads[Node] = // ...
implicit val edgeReads: Reads[Edge] = // ...
Apart from that, there is the first step of a Read for the JsArray itself:
implicit val elementSeqReads = Reads[Seq[Element]](json => json match {
case JsArray(elements) => ???
case _ => JsError("Invalid JSON data (not a json array)")
})
The part with the question marks is responsible for creating a JsSuccess(Seq(node1, edge1, ...) if all elements of the JsArray are valid nodes and edges or a JsError if this is not the case.
However, I'm not sure how to do this in an elegant way.
The logic to distinguish between nodes and edges could look like this:
def hasType(item: JsValue, elemType: String) =
(item \ "elemType").asOpt[String] == Some(elemType)
val result = elements.map {
case n if hasType(n, "node") => // use nodeReads
case e if hasType(e, "edge") => // use edgeReads
case _ => JsError("Invalid element type")
}
The thing is that I don't know how to deal with nodeReads / edgeReads at this point. Of course I could call their validate method directly, but then result would have the type Seq[JsResult[Element]]. So eventually I would have to check if there are any JsError objects and delegate them somehow to the top (remember: one invalid array element should lead to a JsError overall). If there are no errors, I still have to produce a JsSuccess[Seq[Element]] based on result.
Maybe it would be a better idea to avoid the calls to validate and work temporarily with Read instances instead. But I'm not sure how to "merge" all of the Read instances at the end (e.g. in simple case class mappings, you have a bunch of calls to JsPath.read (which returns Read) and in the end, validate produces one single result based on all those Read instances that were concatenated using the and keyword).
edit: A little bit more information.
First of all, I should have mentioned that the case classes Node and Edge basically have the same structure, at least for now. At the moment, the only reason for separate classes is to gain more type safety.
A JsValue of an element has the following JSON-representation:
{
"id" : "aet864t884srtv87ae",
"type" : "node", // <-- type can be 'node' or 'edge'
"name" : "rectangle",
"attributes": [],
...
}
The corresponding case class looks like this (note that the type attribute we've seen above is not an attribute of the class - instead it's represented by the type of the class -> Node).
case class Node(
id: String,
name: String,
attributes: Seq[Attribute],
...) extends Element
The Read is as follows:
implicit val nodeReads: Reads[Node] = (
(__ \ "id").read[String] and
(__ \ "name").read[String] and
(__ \ "attributes").read[Seq[Attribute]] and
....
) (Node.apply _)
everything looks the same for Edge, at least for now.
Try defining elementReads as
implicit val elementReads = new Reads[Element]{
override def reads(json: JsValue): JsResult[Element] =
json.validate(
Node.nodeReads.map(_.asInstanceOf[Element]) orElse
Edge.edgeReads.map(_.asInstanceOf[Element])
)
}
and import that in scope, Then you should be able to write
json.validate[Seq[Element]]
If the structure of your json is not enough to differentiate between Node and Edge, you could enforce it in the reads for each type.
Based on a simplified Node and Edge case class (only to avoid any unrelated code confusing the answer)
case class Edge(name: String) extends Element
case class Node(name: String) extends Element
The default reads for these case classes would be derived by
Json.reads[Edge]
Json.reads[Node]
respectively. Unfortunately since both case classes have the same structure these reads would ignore the type attribute in the json and happily translate a node json into an Edge instance or the opposite.
Lets have a look at how we could express the constraint on type all by itself :
def typeRead(`type`: String): Reads[String] = {
val isNotOfType = ValidationError(s"is not of expected type ${`type`}")
(__ \ "type").read[String].filter(isNotOfType)(_ == `type`)
}
This method builds a Reads[String] instance which will attempt to find a type string attribute in the provided json. It will then filter the JsResult using the custom validation error isNotOfType if the string parsed out of the json doesn't matched the expected type passed as argument of the method. Of course if the type attribute is not a string in the json, the Reads[String] will return an error saying that it expected a String.
Now that we have a read which can enforce the value of the type attribute in the json, all we have to do is to build a reads for each value of type which we expect and compose it with the associated case class reads. We can used Reads#flatMap for that ignoring the input since the parsed string is not useful for our case classes.
object Edge {
val edgeReads: Reads[Edge] =
Element.typeRead("edge").flatMap(_ => Json.reads[Edge])
}
object Node {
val nodeReads: Reads[Node] =
Element.typeRead("node").flatMap(_ => Json.reads[Node])
}
Note that if the constraint on type fails the flatMap call will be bypassed.
The question remains of where to put the method typeRead, in this answer I initially put it in the Element companion object along with the elementReads instance as in the code below.
import play.api.libs.json._
trait Element
object Element {
implicit val elementReads = new Reads[Element] {
override def reads(json: JsValue): JsResult[Element] =
json.validate(
Node.nodeReads.map(_.asInstanceOf[Element]) orElse
Edge.edgeReads.map(_.asInstanceOf[Element])
)
}
def typeRead(`type`: String): Reads[String] = {
val isNotOfType = ValidationError(s"is not of expected type ${`type`}")
(__ \ "type").read[String].filter(isNotOfType)(_ == `type`)
}
}
This is actually a pretty bad place to define typeRead :
- it has nothing specific to Element
- it introduces a circular dependency between the Elementcompanion object and both Node and Edge companion objects
I'll let you think up of the correct location though :)
The specification proving it all works together :
import org.specs2.mutable.Specification
import play.api.libs.json._
import play.api.data.validation.ValidationError
class ElementSpec extends Specification {
"Element reads" should {
"read an edge json as an edge" in {
val result: JsResult[Element] = edgeJson.validate[Element]
result.isSuccess should beTrue
result.get should beEqualTo(Edge("myEdge"))
}
"read a node json as an node" in {
val result: JsResult[Element] = nodeJson.validate[Element]
result.isSuccess should beTrue
result.get should beEqualTo(Node("myNode"))
}
}
"Node reads" should {
"read a node json as an node" in {
val result: JsResult[Node] = nodeJson.validate[Node](Node.nodeReads)
result.isSuccess should beTrue
result.get should beEqualTo(Node("myNode"))
}
"fail to read an edge json as a node" in {
val result: JsResult[Node] = edgeJson.validate[Node](Node.nodeReads)
result.isError should beTrue
val JsError(errors) = result
val invalidNode = JsError.toJson(Seq(
(__ \ "type") -> Seq(ValidationError("is not of expected type node"))
))
JsError.toJson(errors) should beEqualTo(invalidNode)
}
}
"Edge reads" should {
"read a edge json as an edge" in {
val result: JsResult[Edge] = edgeJson.validate[Edge](Edge.edgeReads)
result.isSuccess should beTrue
result.get should beEqualTo(Edge("myEdge"))
}
"fail to read a node json as an edge" in {
val result: JsResult[Edge] = nodeJson.validate[Edge](Edge.edgeReads)
result.isError should beTrue
val JsError(errors) = result
val invalidEdge = JsError.toJson(Seq(
(__ \ "type") -> Seq(ValidationError("is not of expected type edge"))
))
JsError.toJson(errors) should beEqualTo(invalidEdge)
}
}
val edgeJson = Json.parse(
"""
|{
| "type":"edge",
| "name":"myEdge"
|}
""".stripMargin)
val nodeJson = Json.parse(
"""
|{
| "type":"node",
| "name":"myNode"
|}
""".stripMargin)
}
if you don't want to use asInstanceOf as a cast you can write the
elementReads instance like so :
implicit val elementReads = new Reads[Element] {
override def reads(json: JsValue): JsResult[Element] =
json.validate(
Node.nodeReads.map(e => e: Element) orElse
Edge.edgeReads.map(e => e: Element)
)
}
unfortunately, you can't use _ in this case.
I have following (simplified) structure:
case class MyKey(key: String)
case class MyValue(value: String)
Let's assume that I have Play JSON formatters for both case classes.
As an example I have:
val myNewMessage = collection.immutable.Map(MyKey("key1") -> MyValue("value1"), MyKey("key2") -> MyValue("value2"))
As a result of following transformation
play.api.libs.json.Json.toJson(myNewMessage)
I'm expecting something like:
{ "key1": "value1", "key2": "value2" }
I have tried writing the formatter, but somehow I can not succeed:
implicit lazy val mapMyKeyMyValueFormat: Format[collection.immutable.Map[MyKey, MyValue]] = new Format[collection.immutable.Map[MyKey, MyValue]] {
override def writes(obj: collection.immutable.Map[MyKey, MyValue]): JsValue = Json.toJson(obj.map {
case (key, value) ⇒ Json.toJson(key) -> Json.toJson(value)
})
override def reads(json: JsValue): JsResult[collection.immutable.Map[MyKey, MyValue]] = ???
}
I have no idea how to write proper reads function. Is there any simpler way of doing it? I'm also not satisfied with my writes function.
Thx!
The reason the writes method is not working is because you're transforming the Map[MyKey, MyValue] into a Map[JsValue, JsValue], but you can't serialize that to JSON. The JSON keys need to be strings, so you need some way of transforming MyKey to some unique String value. Otherwise you'd be trying to serialize something like this:
{"key": "keyName"} : {"value": "myValue"}
Which is not valid JSON.
If MyKey is as simple as stated in your question, this can work:
def writes(obj: Map[MyKey, MyValue]): JsValue = Json.toJson(obj.map {
case (key, value) => key.key -> Json.toJson(value)
}) // ^ must be a String
Play will then know how to serialize a Map[String, MyValue], given the appropriate Writes[MyValue].
But I'm not certain that's what you want. Because it produces this:
scala> Json.toJson(myNewMessage)
res0: play.api.libs.json.JsValue = {"key1":{"value":"value1"},"key2":{"value":"value2"}}
If this is the output you want:
{ "key1": "value1", "key2": "value2" }
Then your Writes should look more like this:
def writes(obj: Map[MyKey, MyValue]): JsValue = {
obj.foldLeft(JsObject(Nil)) { case (js, (key, value)) =>
js ++ Json.obj(key.key -> value.value)
}
}
Which produces this:
scala> writes(myNewMessage)
res5: play.api.libs.json.JsValue = {"key1":"value1","key2":"value2"}
Reads are easy so long as the structure of MyKey and MyValue are the same, otherwise I have no idea what you'd want it to do. It's very dependent on the actual structure you want. As is, I would suggest leveraging existing Reads[Map[String, String]] and transforming it to the type you want.
def reads(js: JsValue): JsResult[Map[MyKey, MyValue]] = {
js.validate[Map[String, String]].map { case kvMap =>
kvMap.map { case (key, value) => MyKey(key) -> MyValue(value) }
}
}
It's hard to see much else without knowing the actual structure of the data. In general I stay away from having to serialize and deserialize Maps.
I cannot figure out if there is a way to transform a JSON fragment (as a String) in a Map[Symbol,Any] using play.api.libs.json library, where Any could be a Int, a Double, a String or a nested Map[Symbol,Any].
Can anybody give me a hint to get this?
JsObject.fieldSet will give you a Set[(String, JsValue)] that you can transform into a Map[Symbol, Any]. You will have to pattern match on all possible subclasses of JsValue and transform each to the type you want.
For example, something like this:
Json.parse(text) match {
case js: JsObject =>
js.fieldSet.map {
case (key, value) => Symbol(key) -> transform(value)
}.toMap
case x => throw new RuntimeException(s"Expected object json but got $text")
}
def transform(jsValue): Any = jsValue match {
case JsNumber(value) => value.toDouble
...ect...
}