Im trying to implement validation by using this article ScalaJsonCombinators
Basically i want to get the value if exist and if not return null
val nextPage: JsResult[JsValue] = (ch.\("paging").\("next")).validate[JsValue]
val nextUrl: String = nextPage match {
case s: JsSuccess[String] => s.get
case e: JsError => null
}
I have tow issue`s
the first is a warning
Warning:(99, 19) non-variable type argument String in type pattern play.api.libs.json.JsSuccess[String] is unchecked since it is eliminated by erasure
case s: JsSuccess[String] => s.get
^
the second is an error because the string is a URI with special
characters im getting a scheme error
val nextPage: JsResult[JsValue] = (ch.("paging").("next")).validate[JsValue]
val nextUrl: String = nextPage match {
case s: JsSuccess[String] => s.toString
case e: JsError => null
}
java.lang.IllegalArgumentException: Illegal character in scheme name at index 9: JsSuccess(https://graph.facebook.com/v2.2/396697410351933/feed?limit=200&access_token=623501864400497%7CGumAg3k6Eu
What will be to correct way to validate this element without using serialization.??
thanks,
miki
You're validating incorrectly. calling validate[JsValue] on a JsValue is meaningless, because it will always result in JsSuccess[JsValue]. The other problem is you're then trying to pattern match a JsSuccess[String] from a JsSuccess[JsValue], which you cannot do for many reasons. One, the type argument is erased at runtime as noted by the compiler warning. And two, a JsSuccess[String] can't be a JsSuccess[JsValue], the types aren't related.
What you really need is validate[String].
Probably impossible to debug unless you provide the relevant JSON.
There's a more elegant way of doing this, though. Since you don't seem to care about the failures (you're discarding them for null), you can just use asOpt. Here is where I say that if there's a chance there is no next, then nextUrl should be Option[String] instead of String, and you should never ever ever use null in Scala.
val nextUrl: Option[String] = (ch \ "paging" \ "next").asOpt[String]
If you for some reason must use null:
val nextUrl: String = (ch \ "paging" \ "next").asOpt[String].getOrElse(null)
Related
I'm working on some largeish JSON output. We've got a bunch of tests to check the output. We've created the tests by having a copy of the JSON on disk, performing a Json.parse() on the InputStream and comparing that to the JsObject we have build in memory.
This worked well until I started converting some of our Writes to using the functional syntax (i.e. instead of overriding the writes method in the trait, use the builders).
Suddenly, tests started failing: complaining about null fields that were not equal.
Apparently, when using functional syntax, an Option[String] will convert to a JsString(null) instead of JsNull. This is not noticeable In the stringified version.
Consider the following snippet, using
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.7.4"
import play.api.libs.json._
import play.api.libs.functional.syntax._
object FooBar {
case class Foo(option: Option[String])
def main(args: Array[String]): Unit = {
val classic: OWrites[Foo] = new OWrites[Foo] {
override def writes(f: Foo): JsObject = Json.obj("foo" -> f.option)
}
val dsl: OWrites[Foo] = (__ \ "foo").write[String].contramap(foo => foo.option.orNull)
val json_classic = Json.toJsObject(Foo(None))(classic)
val json_dsl = Json.toJsObject(Foo(None))(dsl)
val json_parse = Json.parse("""{"foo":null}""")
val string_classic = Json.prettyPrint(json_classic)
val string_dsl = Json.prettyPrint(json_dsl)
println(
s"""Result is:
|json_dsl == json_classic : ${json_dsl == json_classic} // (expect true)
|json_dsl == json_parse : ${json_dsl == json_parse} // (expect true)
|json_classic == json_parse : ${json_classic == json_parse} // (expect true)
|string_classic == string_dsl : ${string_classic == string_dsl} // (expect true)
|""".stripMargin)
println(s"classic:\n$string_classic")
println(s"dsl:\n$string_dsl")
}
}
Actual output is
Result is:
json_dsl == json_classic : false // (expect true)
json_dsl == json_parse : false // (expect true)
json_classic == json_parse : true // (expect true)
string_classic == string_dsl : true // (expect true)
classic:
{
"foo" : null
}
dsl:
{
"foo" : null
}
When debugging, you'll see that the classic create a wrapper object with a Tuple ("foo", JsNull), whereas the dsl creates a wrapper with a Tuple ("foo", JsString(null)).
It seems that the intended way of the dsl is to use writeNullable in this case, but it feels odd that it works this way.
I'd either expect JsString(null) == JsNull to be true, or that the dsl would catch the null value and prevent a JsString from being created.
Am I doing something totally misguided?
I would just rewrite to .writeNullable[String], which will remove the field from the JSON, but we have a schema in place that requires the field to be present:
...
"properties": {
...
"foo": {
"oneOf": [
{"type": "string"},
{"type": "null"}
]
},
...
}
...
"required": [ "foo" ],
This is part of an API, so changing it will take time.
To clarify: String representation is correct in all cases. I'm only interested in the in-memory representation of the JsValue so I can use its equality during testing.
I think what you want is this:
val dsl: OWrites[Foo] = (__ \ "foo").writeOptionWithNull[String]
.contramap(_.option)
Or, if using an older play-json version that doesn't have writeOptionWithNull:
val dsl: OWrites[Foo] = (__ \ "foo").write[JsValue]
.contramap(_.option match {
case None => JsNull
case Some(string) => JsString(string)
})
Note, play.api.libs.json.JsNull, and null, are two completely different and unrelated concepts, and should never be confused or mixed. The convention across the whole Scala ecosystem in Scala-only APIs is to never, ever use null, to the extent where most libraries simply pretend that it doesn't even exist. So, you won't find many Scala libraries that do null checks, they just assume everything is non null, and the moment you start using null, you're in the wild west of untested and unsupported edge cases. The only time you use or deal with nulls is if working with a Java API, because they use nulls, and the convention there is to wrap anything that can produce a null in Option as early as possible, and unwrap Option using orNull as late as possible, so that at least internally in the Scala code that doesn't directly interface to Java code, everything is using Option, rather than null. But play-json is designed for Scala use only, so like the rest of the Scala-only ecosystem, it just assumes null doesn't exist.
Of course, in json, null does exist, and there may be valid reasons to use it (especially when integrating with other systems that require it). So play-json does model it, but it models it in a very strongly typed way - nothing will ever automatically go from being null in Scala to being null in JSON, it's always very explicit, this is in line with Scala being strongly typed. Also, use of null in JSON doesn't tend to be that common, so most of the default methods (ie, the writeNullable method) map Option to a non-existent value, so you have to be a little more explicit when you want to write a JSON null, as shown above.
I cannot reproduce your tests in REPL with Play-JSON 2.7.4:
import play.api.libs.json._
case class Foo(option: Option[String])
val classic: OWrites[Foo] = new OWrites[Foo] {
override def writes(f: Foo): JsObject = Json.obj("foo" -> f.option)
}
val dsl: OWrites[Foo] = (__ \ "foo").write[String].contramap(foo => foo.option.orNull)
val json_classic = Json.toJsObject(Foo(None))(classic)
val json_dsl = Json.toJsObject(Foo(None))(dsl)
val json_classic = Json.toJsObject(Foo(None))(classic)
// json_classic: play.api.libs.json.JsObject = {"foo":null}
val json_dsl = Json.toJsObject(Foo(None))(dsl)
// json_dsl: play.api.libs.json.JsObject = {"foo":null}
The inequality between json_classic and json_dsl is something else, but the JSON representation is consistent in both cases, even if the foo.option.orNull is unsafe/weird for me.
On the otherwise, if you want to consider "null" as null, you can override the default Reads[String] where this specific behaviour is wanted.
scala> val legacyStrReads: Reads[Option[String]] =
| Reads.optionWithNull(Reads.StringReads).map {
| case Some("null") => None
| case other => other
| }
legacyStrReads: play.api.libs.json.Reads[Option[String]] = play.api.libs.json.Reads$$anon$6#138decb1
scala> Json.toJson("null").validate(legacyStrReads)
res9: play.api.libs.json.JsResult[Option[String]] = JsSuccess(None,)
First of all - sorry for stupid question.
I got some Json string from DB and want to parse all of them with json4s:
val df = sqlContext.sql("SELECT * FROM analytic.test").repartition(22)
val df_base = df.map(f => {
implicit val formats = DefaultFormats
val jsonString = f(5).toString
val tempJSON = parse(jsonString)
val mainJsonArray = tempJSON \ "events"
(
f(2).toString,
makeEventArray(mainJsonArray)
)
}).cache()
All good, i got Json's, but sometimes in DB occurs some failed Json, that take me to error :
com.fasterxml.jackson.core.JsonParseException: Unexpected end-of-input: was expecting closing '"' for name
First question - How can I evade this row with corrupt Json and continue my program?
I trying surround parse with try\catch, but in this case:
var tempJSON = json4s.JsonAST.JValue
try {
tempJSON = parse(f(5).toString)
} catch {
case e: Exception => println("Error on JSON parser. " + e )
}
But taking error:
Error:(51, 25) type mismatch;
found: org.json4s.JValue (which expands to) org.json4s.JsonAST.JValue
required: org.json4s.JsonAST.JValue.type tempJSON = parse(f(5).toString)
^
Second question - How to declare tempJson right?
Or I must validate Json before parse? How?
You can use Try for it:
val tempJSON = Try(parse(f(5).toString))
So now, you can match it:
tempJSON match {
case Success(yourValue) => ???
case Failure(exception) => println(exception.getMessage)
}
Or, if you don't need the exception, you would convert it to Option:
tempJSON.toOption
You'll get None or Some(value).
I don't know json4s but it probably exists, like in Play Json, a validate function returning something like a JsError or a JsSuccess.
Otherwise, an other way to go is to return an Option[JValue] (if you don't want to deal with exceptions), i.e. :
def function: Option[JValue] = {
try {
Option(parse(f(5).toString))
} catch {
case e: Exception => None
}
}
If you want to catch all parsing errors and silently skip them, which is risky and ill advised in most cases.
You can simply do:
df.flatMap(f => Try(do something).toOption)
You may however prefer, earlier validation preventing this to begin with, catching a more specific error only, logging errors etc.
I'm trying to check JsValue object in my Actor using play framework 2.2.2. When I try to use validate method, I receive exception not a result object:
try {
val result = data.validate[EventConfig]
Logger.debug("Result: "+result")
} catch {
case e =>
Logger.error("Exception: "+e)
}
Here is this exception:
Exception: play.api.libs.json.JsResultException: JsResultException(errors:List((,List(ValidationError(error.expected.jsnumber,WrappedArray())))))
Why is this happening, and how should I use validate method?
====== Update
I was using such Reads implementation:
implicit val EventConfig_reads = new Reads[EventConfig] {
def reads(json: JsValue): JsResult[EventConfig] = {
JsSuccess(new
EventConfig((json \ ConfigEventAttrs.PARAM).as[Int],
(json \ ConfigEventAttrs.PERIOD).as[Int],
(json \ ConfigEventAttrs.THRESHOLD).as[Int],
(json \ ConfigEventAttrs.TOGGLE).as[Boolean]))
}
}
The solution is to add catch clause:
implicit val EventConfig_reads = new Reads[EventConfig] {
def reads(json: JsValue): JsResult[EventConfig] = {
try {
JsSuccess(new
EventConfig((json \ ConfigEventAttrs.PARAM).as[Int],
(json \ ConfigEventAttrs.PERIOD).as[Int],
(json \ ConfigEventAttrs.THRESHOLD).as[Int],
(json \ ConfigEventAttrs.TOGGLE).as[Boolean]))
} catch {
case e: JsResultException =>
JsError(e.errors)
}
}
}
That is not the proper way to use validate. I don't think the documentation highlights it's importance as much as it should, but it's explained here, in the section called Using Validation.
data.validate[EventConfig] returns JsResult and not EventConfig. The preferred way to deal with errors is to fold the result:
data.validate[EventConfig].fold(
error => {
// There were validation errors, handle them here.
},
config => {
// `EventConfig` has validated, and is now in the scope as `config`, proceed as usual.
}
)
Let's examine this a bit. The signature if fold on a JsResult is as follows:
fold[X](invalid: (Seq[(JsPath, Seq[ValidationError])]) ⇒ X, valid: (A) ⇒ X): X
It accepts two functions as arguments that both return the same type of result. The first function is a Seq[(JsPath, Seq[ValidationError])]) => X. In my code above, error has the type Seq[(JsPath, Seq[ValidationError])]), which is essentially just a sequence of json paths tupled with their validation errors. Here you can dissect these errors and return the appropriate error messages accordingly, or do whatever else you may need to on failure.
The second function maps A => X, where A is the type JsResult has been validated as, in your case EventConfig. Here, you'll be able to handle your EventConfig type directly.
Causing and catching exceptions is not the way to handle this (and rarely is), as you will lose all of the accumulated validation errors.
Edit: Since the OP has updated his question with additional information regarding his defined Reads.
The problem with the Reads defined there is that they're using as[T]. When calling as, you're trying to force the given json path to type T, which will throw an exception if it cannot. So as soon as you reach the first validation error, an exception is thrown and you will lose all subsequent errors. Your use case is relatively simple though, so I think it would be better to adopt a more modern looking Reads.
import play.api.libs.json._
import play.api.libs.functional.syntax._
case class EventConfig(param: Int, period: Int, threshold: Int, toggle: Boolean)
object EventConfig {
implicit val jsonReads: Reads[EventConfig] = (
(__ \ ConfigEventAttrs.PARAM).read[Int] and
(__ \ ConfigEventAttrs.PERIOD).read[Int] and
(__ \ ConfigEventAttrs.THRESHOLD).read[Int] and
(__ \ ConfigEventAttrs.TOGGLE).read[Boolean]
)(EventConfig.apply _)
}
This is much more compact, and the use of the functional syntax will accumulate all of the validation errors into the JsResult, as opposed to throwing exceptions.
Edit 2: To address the OP's need for a different apply method.
If the parameters you're using the build an object from JSON differ from those of your case class, define a function to use for the JSON Reads instead of EventConfig.apply. Supposing your EventConfig is really like this in JSON:
(time: Long, param: Int)
But instead you want it to be like this:
case class EventConfig(time: Date, param: Int)
Define a function to create an EventConfig from the original parameters:
def buildConfig(time: Long, param: Int) = EventConfig(DateUtils.timeSecToDate(time), param)
Then use buildConfig instead of EventConfig.apply in your Reads:
implicit val jsonReads: Reads[EventConfig] = (
(__ \ "time").read[Long] and
(__ \ "param").read[Int]
)(buildConfig _)
I shortened this example, but buildConfig can be any function that returns EventConfig and parameters match those of the JSON object you're trying to validate.
Validating depends on your Reads method, and I've had an issue there. I should just catch this exception in my reads.
I have two functions that are similar. One returns a String when successful and the second returns an Option[String] when successful. How can write this in a more elegant way, perhaps calling the first function in the second function? Thanks!
private def readString(j: JsValue, key: String): Validation[String, String] = {
j \ key match {
case j: JsString =>
Success(j.value)
case j: JsUndefined =>
Failure(s"Missing field $key")
case j: JsValue =>
Failure(s"Field $key value is not a string")
}
}
private def readStringOpt(j: JsValue, key: String): Validation[String, Option[String]] = {
(j \ key).as[JsValue] match {
case j: JsString =>
j.value.some.success
case j: JsUndefined =>
None.success
case x =>
Failure(s"Field $key value is not a string")
}
}
First, the .as[JsValue] in your second method isn't really necessary—all it does is look up the Reads instance for JsValue, which just passes through its argument and never fails.
If you want to go the route you suggest above (defining the Opt-less version in terms of the other), Scalaz provides some slightly more concise syntax:
def readString(j: JsValue, key: String): Validation[String, String] =
readStringOpt(j, key).flatMap(_.toSuccess(s"Missing field $key"))
This will give you a deprecation warning in recent versions of Scalaz, however, since Validation does not have a monad instance and its flatMap is kind of a lie. This means you have two options (apart from ignoring the deprecation warning): you can switch to \/, which is monadic, or you can use a fold instead:
def readString(j: JsValue, key: String): Validation[String, String] =
readStringOpt(j, key).fold(_.failure, _.toSuccess(s"Missing field $key"))
Which is a little more verbose, but it puts you on the right side of the validation-is-not-a-monad gods.
I have a code for extracting Int from JValue that should look exactly the same for multiple JValue subclasses, so I'm trying to avoid repeating myself. However, as it is (see below), scala thinks that j is a generic JValue, and that j.values returns a value of type Values, which, of course, does not have isValidInt and toInt methods.
jvalue \ name match {
case j # (JInt | JDecimal | JDouble) => {
val num = j.values
if (num.isValidInt) num.toInt.success else reportError(name + " is not a valid int")
}
The question is what is the proper way to avoid repetition here? I'm not that crazy about converting a JValue into string, because this code works with json just parsed from a string into an AST. I started thinking about writing wrappers for the three types I need matched and implicit converters from these types into wrappers, and then making a superclass just for these wrappers for use as a pattern, but I am not sure how to pull it off.
And yes, I realize there are multiple similar questions here (such as this and this), but each of them only contains part of the solution at most.
Implement an extractor to get the Int value out of arbitrary json values, then use the extractor in your mattern match.
object JsonInt {
def unapply(json: JValue): Option[Int] = json match {
case JInt(i) if i.isValidInt => Some(i.toInt)
case JDecimal(d) if d.isValidInt => Some(d.toInt)
case JDouble(d) if d.isValidInt => Some(d.toInt)
case _ => None
}
}
jvalue \ name match {
case JsonInt(num) => num.success
case _ => reportError(s"$name is not a valid int")
}
Scala has only limited support for structural typing. Seems like JValue is the lowest common ancestor of these three types. If you want, you may get around it by defining an implicit conversion from JValue to some wrapper class that will have isValidInt method.
If you don't bother using structural typing, you can use asInstanceOf method to be able to use toInt and isValidInt method:
type IntExt = {
def toInt: Int
def isValidInt: Boolean
}
jvalue \ name match {
case j # (_: JInt | _: JDecimal | _: JDouble) =>
val num = j.values.asInstanceOf[IntExt]
if (num.isValidInt) num.toInt.success else reportError(name + " is not a valid int")
}