JSON validation using Scala - json

I am trying to write a scala application for JSON validation. I have a Animals.scala class that defines the following:
case class Animals (id: Int, type: String, targets: String)
object Animals {
implicit val reads: Reads[Animals] = (
(JsPath \ "id").read[Int] and
(JsPath \ "type").read[String] and
(JsPath \ "targets").read[String])(Animals.apply _)
}
I have Application.scala where I have tried to validate an incoming JSON against the case class.
object Application extends Controller {
// action for JSON validation
def validateRequest = Action { implicit request =>
// this will fail if the request body is not a valid json value
val bodyAsJson = request.body.asJson.get
bodyAsJson.validate[Animals] match {
case success: JsSuccess[Animals] => {
val id = success.get.id
Ok("Validation passed! id is "+ id)
}
case JsError(error) => BadRequest("Validation failed!")
}
}
}
And finally here's my JSON input:
{
"id" : 1,
"type" : "domestic",
"targets": {
"AND": [
{
"breed": ["greyhound", "dalmatian"]
},
{
"NOT": {
"color": ["amber", "pale_amber", "black"]
}
},
{
"zipcode": ["90210", "90211"]
}
]
}
}
And I get the following error:
JsError(List((/targets,List(ValidationError(error.expected.jsarray,WrappedArray())))))
I do realize that the error is thrown because targets field is not as simple as a String compared to my JSON. How do I wrap it so that the validation passes? Should I do List[List[String]] or something along those lines?

If you don't care about the structure of targets read it as a JsObject. It will parse any internal structure that way.

Related

Issue with parsing nested JSON values with Lift-JSON

I'm using Scala 2.12 and trying to parse the below JSON file.
{
"comp1": {
"metrics": {
"operation1": {
"alias": "activity_operation",
"weight": 10
},
"operation2": {
"alias": "service_operation",
"weight": 22
}
}
},
"comp2": {
"metrics": {
"operation1": {
"alias": "activity_operation",
"weight": 14
},
"operation4": {
"alias": "service_operation",
"weight": 16
}
}
}
}
I've loaded the json into config variable, defined a case class and trying the below:
case class OperationDetails(alias: String, weight: Int)
for (detail <- (config \ "comp1").children) {
println(detail.extract[OperationDetails])
}
This gives me the error Exception in thread "main" net.liftweb.json.MappingException: No usable value for alias. Did not find value which can be converted into java.lang.String
I can't use `operation1' and retrieve children as operations are random.
I need to retrieve the operation names operation1, operation2, operation4, .. and their respective aliases and weights. Any ideas?
You are missing at least one level of nesting, and possibly also the implicit val formats.
This will print all of the operations. Note the conversion into JObject in order to be able to retrieve field names.
// Setup
case class Operation(alias: String, weight: Int)
implicit val formats = DefaultFormats
// Traversal
val comps: List[JsonAST.JValue] = config.children
for (comp <- comps) {
val metrics:List[JsonAST.JValue] = comp.children
for (metric <- metrics) {
val operations:List[JsonAST.JField] = metric.asInstanceOf[JObject].obj
for (operation <- operations) {
val op = operation.value.extract[Operation]
// Do something here
println(s"${operation.name}:(${op.alias},${op.weight})")
}
}
}
Output:
operation1:(activity_operation,10)
operation2:(service_operation,22)
operation1:(activity_operation,14)
operation4:(service_operation,16)

Restructure a json in scala play

The following code gets the request body and validates and creates a json:
object ValidateDBConfigJson {
implicit val reads: Reads[ValidateDetails] = (
(JsPath \ "name").read[String].filter(JsonValidationError("Invalid name"))(_.length > 0) and
(JsPath \ "email").read[String].filter(JsonValidationError("Invalid email"))(_.length > 0) and
)(ValidateDetails.apply _)
}
def index() = Action { implicit request =>
val bodyAsJson = request.body.asJson.get
bodyAsJson.validate[ValidateDetails] match {
case success: JsSuccess[ValidateDetails] => {
Ok(Json.parse("succeeded!"))
}
case JsError(error) =>
BadRequest(JsError.toJson(error))
}
}
The json looks like this:
{
"obj.name": [
{
"msg": [
"error.expected.jsstring"
],
"args": []
}
],
"obj.email": [
{
"msg": [
"Invalid email"
],
"args": []
}
]
}
I want structured in the following format:
{
"ErrorMessages" :
[
"error.expected.jsstring",
"Invalid email"
]
}
Preamble: when I am parsing JSON using Play I prefer to use case class/objects rather than implicit reads, so this answer will cover that method of doing this. There may be a simpler method of doing this with implicit reads but I am not as familiar with implicit reads.
Firstly, define a case class for everything you will be taking from JSON:
object Input {
case class Err(msg: Seq[String], args: Seq[String])
object Err {
implicit val format: OFormat[Err] = Json.format[Err]
}
case class ValidateDetails(`obj.name`: Seq[Err], `obj.email`: Seq[Err])
object ValidateDetails {
implicit val format: OFormat[ValidateDetails] = Json.format[ValidateDetails]
}
}
Note: Play won't know how to handle user-defined case classes so I've made one for Err as well. The implicit val format: OFormat[ValidateDetails] = Json.format[ValidateDetails] and implicit val format: OFormat[Err] = Json.format[Err] lines are magic and do all of the reads/writes for you.
Next, define a case class for your output JSON and define a function which will turn your input case class into your output one:
object Output {
case class OutputJson(`ErrorMessages`: Seq[String])
object OutputJson {
implicit val format: OFormat[OutputJson] = Json.format[OutputJson]
}
// take msg Seq from name & email and add together into single Seq
def inputToOutput(input: Input.ValidateDetails): OutputJson = {
OutputJson(input.`obj.name`.flatMap(_.msg) ++ input.`obj.email`.flatMap(_.msg))
}
}
Finally, put this into a method which maps to a POST route in your routes file:
def index() = Action { implicit request =>
val bodyAsJson = request.body.asJson.get
bodyAsJson.validate[Input.ValidateDetails] match {
case success: JsSuccess[Input.ValidateDetails] =>
// turn the JSON into the Output case class and parse that as JSON
val output: JsValue = Json.toJson(Output.inputToOutput(success.value))
Ok(output)
case JsError(error) =>
BadRequest(JsError.toJson(error))
}
}
Now, if you run the Play app on port 9000 and POST to http://localhost:9000/ with the below JSON body...
{
"obj.name": [
{
"msg": [
"error.expected.jsstring"
],
"args": []
}
],
"obj.email": [
{
"msg": [
"Invalid email"
],
"args": []
}
]
}
...the output will be:
{
"ErrorMessages": [
"error.expected.jsstring",
"Invalid email"
]
}
I hope this answers your question.

Parsing/extracting information from JSON in Scala

I'm trying to extract (timestamp, value) tuples from this third party JSON message:
{
"data": {
"1496845320000": [
0.14
],
"1496845380000": [
0.14
],
"1496845560000": [
0.14
],
"1497013740000": [
"undef"
],
"1497013800000": [
"undef"
],
"1497013860000": [
"undef"
]
},
"status": "ok"
}
This is how I'm extracting the JSON entity:
val batchReadings = http.singleRequest(httpRequest) map {
case HttpResponse(StatusCodes.OK, headers, entity, _) =>
entity.dataBytes.runFold(ByteString(""))(_ ++ _).foreach { body =>
JSON.parseFull(body.utf8String) match {
case Some(e: Map[String, Any]) =>
e.get("status") match {
case Some("ok") => println("ok!")
e.get("data") match {
case Some(f: Map[String, List[Any]]) =>
println("entered second case")
new EfergyReadingsData(f.toList).getReadings
}
case _ => log.info("EfergyReaderWorkerActor: Received unknown message\n" + body.utf8String)
}
}
}
case HttpResponse(code, _, entity, _) =>
//entity.toStrict(100, materializer) N.B. Not having this, will freeze after 4 calls!
s"Response code: $code"
case unknown =>
s"unknown response: $unknown"
}
And this is the EfergyReadingsData class:
class EfergyReadingsData(readings: List[(String, List[Any])]) {
def getReadings: List[(Long, Double)] = {
println(readings)
// List((1123, 34.4))
val filteredReadings = readings filter {
f =>
f._2 match {
case t:List[Any] =>
t.head match {
case Double => true
case _ => false
}
}
}
val theRead = filteredReadings map {
p => (p._1.toLong, p._2.head)
}
println(theRead)
List((1123, 34.4))
}
}
As you can see I'm not returning the List of tuples that I'm supposed to in getReadings, I just created a list to be returned so I could test the code.
Besides not being able to filter the list to remove only the undefentries (the filteredReadings is empty), the first obvious problem with this approach is that I'm pattern matching for types inside List and Map and therefore getting a warning about those being unchecked because they are eliminated by erasure.
Although not represented above for simplicity, I tried to mitigate this by using case classes for the Lists:
sealed abstract class AnyReading {
val value: List[Any]
}
case class SuccessfulReading(override val value: List[Double]) extends AnyReading
case class FailedReading(override val value: List[String]) extends AnyReading
So, to conclude, what is the best way to go about this? I also thought of creating few case classes to represent the the JSON structure and try to deserialise the JSON message and then filter to remove the undef values, and ultimately get the list of tuples I wanted.
Thank you in advance for your help.

Test response is a JsonArray -- Play framework 2.4.2 Spec 2 testing

I'm trying to test bellow using Play 2.4.2 , Spec 2 ,
" test response Returns a json Array" in new WithApplication {
val response = route(FakeRequest(GET, "/myservice/xx")).get
// ??? test response is a json array
}
What would be the way to test this scenario ?
Here is a possibility
Controller
#Singleton
class BarryController extends Controller{
def barry = Action { implicit request =>
val json: JsValue = Json.parse("""
{
"residents" : [ {
"name" : "Fiver",
"age" : 4,
"role" : null
}, {
"name" : "Bigwig",
"age" : 6,
"role" : "Owsla"
} ]
}
""")
Ok(json)
}
}
Test
import org.specs2.mutable._
import play.api.mvc._
import play.api.test.FakeRequest
import play.api.test.Helpers._
import play.api.test.WithApplication
import controllers._
import play.api.libs.json._
class BarryControllerSpec extends Specification {
"controllers.BarryController" should {
val expectedJson: JsValue = Json.parse("""
{
"residents" : [ {
"name" : "Fiver",
"age" : 4,
"role" : null
}, {
"name" : "Bigwig",
"age" : 6,
"role" : "Owsla"
} ]
}
""")
"respond with JsArray for /barry" in new WithApplication {
val result = new controllers.BarryController().barry()(FakeRequest())
status(result) must equalTo(OK)
contentType(result) must equalTo(Some("application/json"))
//testing class is JsArray. The .get is necessary to get type out of JsLookupResult/JsDefined instance
(contentAsJson(result) \ "residents").get must haveClass[JsArray]
//testing JSON is equal to expected
contentAsJson(result) must equalTo(expectedJson)
//test an attribute in JSON
val residents = (contentAsJson(result) \ "residents").get
(residents(0) \ "age").get must equalTo(JsNumber(4))
}
}
}
Hopefully this gives you some ideas on what you can test or what you might want to do.

Cryptic Spray Json error message

I'm receiving the following error message when trying to parse some json:
[info] The future returned an exception of type: spray.httpx.PipelineException, with message:
Vector("eba760a81b177051b0520418b4e10596955adb98196c15367a2467ab66a19b5c", 600, "AN51SPP6iZBHFJ3aux1jtn6MMMD13Gh3t7", 500,
["1BXVXP82f7x9YWdWuCaCYwad8ZoYayyRYt"], "76a91473758c13a91699376abb8fe76931bdd9bdc04ee388ac", false)
(of class scala.collection.immutable.Vector). (AddressUnspentTXORequestTest.scala:14)
and I'm not really sure what that means. Here is the piece of json that I am trying to parse:
[
{
"transaction_hash": "eba760a81b177051b0520418b4e10596955adb98196c15367a2467ab66a19b5c",
"output_index": 1,
"value": 600,
"asset_id": "AN51SPP6iZBHFJ3aux1jtn6MMMD13Gh3t7",
"asset_quantity": 500,
"addresses": [
"1BXVXP82f7x9YWdWuCaCYwad8ZoYayyRYt"
],
"script_hex": "76a91473758c13a91699376abb8fe76931bdd9bdc04ee388ac",
"spent": false,
"confirmations": 31674
},
{
"transaction_hash": "1f9f6224bee8813135aba622693c78a33b3460e4efdb340174f87fdd8c9d4148",
"output_index": 1,
"value": 600,
"asset_id": "AS6tDJJ3oWrcE1Kk3T14mD8q6ycHYVzyYQ",
"asset_quantity": 200000,
"addresses": [
"1BXVXP82f7x9YWdWuCaCYwad8ZoYayyRYt"
],
"script_hex": "76a91473758c13a91699376abb8fe76931bdd9bdc04ee388ac",
"spent": false,
"confirmations": 35895
}
]
and here is the case class that I am trying to parse it into:
case class UnspentTXO(transaction_hash: String, output_index: Int, value: Long,
asset_id: Option[String], asset_quantity: Option[Long], addresses: List[BitcoinAddress],
script_hex: String, spent: Boolean)
The method initiating the request is here :
def getUnspentTXOs(address: Address): Future[List[UnspentTXO]] = {
val pipeline: HttpRequest => Future[List[UnspentTXO]] =
sendReceive ~> unmarshal[List[UnspentTXO]]
pipeline(Get(host + path + address.value + "/unspents"))
}
and finally this is how I am parsing that Json Request:
override def read(value: JsValue): UnspentTXO = {
val Seq(transaction_hash, output_index, locked_satoshies, asset_id, asset_quantity, addresses, script_hex, spent) =
value.asJsObject.getFields("transaction_hash", "value", "asset_id", "asset_quantity", "addresses", "script_hex", "spent")
val assetId = asset_id match {
case JsString(s) => Some(s)
case JsNull => None
case _ => throw new RuntimeException("Asset id should be of type JsString or JsNull, got something else")
}
val assetQuantity = asset_quantity match {
case JsNumber(n) => Some(n.toLong)
case JsNull => None
case _ => throw new RuntimeException("Asset quantity should be JsNull or a JsNumber")
}
// convert JsArray to List[ BitcoinAdress ]
val addressList = addresses match {
case ja: JsArray => {
ja.elements.toList.map( e => BitcoinAddress(e.convertTo[String]))
}
case _ => throw new RuntimeException("address list should be of type JsArray, got something else")
}
UnspentTXO(transaction_hash.convertTo[String], output_index.convertTo[Int], locked_satoshies.convertTo[Long],
assetId, assetQuantity, addressList,
script_hex.convertTo[String], spent.convertTo[Boolean])
}
I think the problem might be that the request is returning a json array instead of just a JSON object, so I am not sure if I am handling that correctly inside of my getUnspentTXOs. The error message seems to be very vague. It seems that spray is trying to wrap the json fields inside of a Vector instead of inside of a UnspentTXO case class. I'm not sure of why this is happening though.
You can not just call convertTo[ Option[ Long ] ] and convertTo[ List [ BitcointAddress ] ].
This is how convertTo is defined,
def convertTo[T :JsonReader]: T = jsonReader[T].read(this)
Which means... only types T for which an implicit evidence of typeclass JsonReader[ T ] is available can be used with convertTo.
Unless you provide appropriate implicit typeclass evidence, you will have to specially handle few cases.
Other than this, spray-json is just too minimalistic... so that so JsObject is just a wrapper on top of Map[ String, JsValue] and getFields is defined as,
def getFields(fieldNames: String*): immutable.Seq[JsValue] =
fieldNames.flatMap(fields.get)(collection.breakOut)
Which means... it will just ignore asked fields which are not present in them map, and only return a sequence of JsValue's corresponding to the asked fields which are present.
Hence optional values have to be checked to be present in the map. Or we will have to pattern-match for each possible case ( which can result in a lot of cases).
override def read(value: JsValue): UnspentTXO = {
val jsObject = value.asJsObject
// get only non-optional values here
val Seq( transaction_hash, output_index, locked_satoshies, addresses,
script_hex, spent ) =
jsObject.getFields( "transaction_hash", "output_index", "value", "addresses", "script_hex", "spent" )
// Assuming you have imported spray.json._, simple types will work.
// have to handle options differently
// or 2 optional values would mean 4 patterns-matchings of sequences like above.
// jsObject.fields is just a Map[ String, JsValue ]
val assetId = jsObject.fields.get( "asset_id" ) match {
case Some( JsString( s ) ) => Some( s )
case None => None
}
val assetQuantity = jsObject.fields.get( "asset_quantity" ) match {
case Some( JsNumber( n ) ) => Some( n.toLong )
case None => None
}
// convert JsArray to List[ BitcoinAdress ]
val addressList = addresses match {
case ja : JsArray => {
ja.elements.toList.map( BitcoinAddress( _.convertTo[ String ] ) )
}
}
UnspentTXO( transaction_hash.convertTo[ String ], output_index.convertTo[ Int ],
locked_satoshies.convertTo[ Long ], assetId, assetQuantity,
addressList, script_hex.convertTo[ String ], spent.convertTo[ Boolean ] )
}
Another way to get convertTo[ Option[ T ] ] and convertTo[ List[ T ] ] working is by importing spray.json.DefaultJsonProtocol._ which provides json-formats for most generally used types. But even then, you must have an implicit evidence of typeclass JsonReader[ T ] in your scope.