How to parse an array of json in scala play framework? - json

I have an array of json objects like this
[
{
"events": [
{
"type": "message",
"attributes": [
{
"key": "action",
"value": "withdraw_reward"
},
{
"key": "sender",
"value": "bob"
},
{
"key": "module",
"value": "distribution"
},
{
"key": "sender",
"value": "bob"
}
]
},
{
"type": "credit",
"attributes": [
{
"key": "recipient",
"value": "ross"
},
{
"key": "sender",
"value": "bob"
},
{
"key": "amount",
"value": "100"
}
]
},
{
"type": "rewards",
"attributes": [
{
"key": "amount",
"value": "100"
},
{
"key": "validator",
"value": "sarah"
}
]
}
]
},
{
"events": [
{
"type": "message",
"attributes": [
{
"key": "action",
"value": "withdraw_reward"
},
{
"key": "sender",
"value": "bob"
},
{
"key": "module",
"value": "distribution"
},
{
"key": "sender",
"value": "bob"
}
]
},
{
"type": "credit",
"attributes": [
{
"key": "recipient",
"value": "ross"
},
{
"key": "sender",
"value": "bob"
},
{
"key": "amount",
"value": "100"
}
]
},
{
"type": "rewards",
"attributes": [
{
"key": "amount",
"value": "200"
},
{
"key": "validator",
"value": "Ryan"
}
]
}
]
}
]
How to traverse through the types, check if it's type equals to rewards and then go through the attributes and verify if the validator equals to sarah and fetch the value of the key amount? Pretty new to scala and play framework. Any help would be great. Thanks

You could parse your JSON into a structure of case classes for easier handling and then extract the wanted field like so:
val json =
"""[
{"events":[
{
"type":"message","attributes":[
{"key":"action","value":"withdraw_reward"},
{"key":"sender","value":"bob"},
{"key":"module","value":"distribution"},
{"key":"sender","value":"bob"}
]},
{
"type":"credit","attributes":[
{"key":"recipient","value":"ross"},
{"key":"sender","value":"bob"},
{"key":"amount","value":"100"}
]},
{
"type":"rewards","attributes":[
{"key":"amount","value":"100"},
{"key":"validator","value":"sara"}
]}
]
},
{"events":[
{
"type":"message","attributes":[
{"key":"action","value":"withdraw_reward"},
{"key":"sender","value":"bob"},
{"key":"module","value":"distribution"},
{"key":"sender","value":"bob"}
]},
{
"type":"credit","attributes":[
{"key":"recipient","value":"ross"},
{"key":"sender","value":"bob"},
{"key":"amount","value":"100"}
]},
{
"type":"rewards","attributes":[
{"key":"amount","value":"200"},
{"key":"validator","value":"Ryan"}
]}
]
}
]
"""
case class EventWrapper(events: Seq[Event])
case class KeyValue(key: String, value: String)
case class Event(`type`: String, attributes: Seq[KeyValue])
import play.api.libs.json._
implicit val kvReads: Reads[KeyValue] = Json.reads[KeyValue]
implicit val eventReads: Reads[Event] = Json.reads[Event]
implicit val eventWrapperReads: Reads[EventWrapper] = Json.reads[EventWrapper]
val rewardAmountsValidatedBySara = Json
.parse(json)
.as[Seq[EventWrapper]]
.flatMap {
_.events.collect {
case Event(t, attributes) if t == "rewards" && attributes.contains(KeyValue("validator", "sara")) =>
attributes.collect {
case KeyValue("amount", value) => value
}
}.flatten
}
val amount = rewardAmountsValidatedBySara.head
For your example, rewardAmountsValidatedBySara would yield a List of Strings containing only the String "100". Which you could retrieve (potentially unsafe) with .head as shown above.
Normally you would not do this, as it could throw an exception on an empty List, so it would be better to use .headOption which returns an Option which you can then handle safely.
Note that the implicit Reads are Macros, which automatically translate into Code, that instructs the Play Json Framework how to read the JsValue into the defined case classes, see the documentation for more info.

Related

building url query string using n1ql

using couchbase 5
I need to build a query string from this object
[
{
"_id": 190,
"querystring": [
{
"name": "p1",
"value": "val1"
},
{
"name": "p2",
"value": "val2"
}
]
}
]
the expected output should be
p1=val1&p2=val2
can anyone help here?
after few attempts I think I got closer to the solution I need.
[
{
"_id": 190,
"res": [
"company_id=$PREFIJO&",
"user_country=$COUNTRY&",
"offer_unique_code=$PIXEL&",
"pub_id=$PUBID&"
]
}
]
now, how can I convert "res" to a concatenated string of all the array elements?
WITH obj AS ({ "_id": 190, "querystring": [ { "name": "p1", "value": "val1" }, { "name": "p2", "value": "val2" } ] })
SELECT obj._id, CONCAT2("&", ARRAY CONCAT2("=",v.name,v.`value`) FOR v IN obj.querystring END) AS res;
Array of objects
WITH objs AS ([{ "_id": 190, "querystring": [ { "name": "p1", "value": "val1" }, { "name": "p2", "value": "val2" } ] },
{ "_id": 191, "querystring": [ { "name": "p3", "value": "val1" }, { "name": "p4", "value": "val2" } ] }
])
SELECT obj._id, CONCAT2("&", ARRAY CONCAT2("=",v.name,v.`value`) FOR v IN obj.querystring END) AS res FROM objs AS obj ;
Older version where CONCAT2() not available, get array of strings (name=val) and do in application or use the following technique. Assume your name/val doesn't have any replace characters.
WITH objs AS ([{ "_id": 190, "querystring": [ { "name": "p1", "value": "val1" }, { "name": "p2", "value": "val2" } ] },
{ "_id": 191, "querystring": [ { "name": "p3", "value": "val1" }, { "name": "p4", "value": "val2" } ] }
])
SELECT obj._id, replace(replace(replace(encode_json(ARRAY CONCAT(v.name,"=",v.`value`) FOR v IN obj.querystring END),"\",\"","&"),"[\"",""),"\"]","") AS res FROM objs AS obj ;
If single document then have ARRAY of objects then use UNNEST
If there is number , convert to string using TO_STR() before CONCAT operation
https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/stringfun.html#fn-str-concat2

Fetching the array name when traversing array items in JSONata

I need to fetch the name of the array while traversing the child array items.
for example, if my input looks like
{"title": [
{
"value": "18724-100",
"locale": "en-GB"
},
{
"value": "18724-5",
"locale": "en-GB"
},
{
"value": "18724-99",
"locale": "fr-FR"
}
]}
I need output as
{
"data": [
{
"locale": "en-GB",
"metadata": [
{
"key": "title",
"value": "18724-100"
},
{
"key": "title",
"value": "18724-5"
}
]
},
{
"locale": "fr-FR",
"metadata": {
"key": "title",
"value": "18724-99"
}
}
]
}
I tried following spec in JSONata
{
"data": title{locale: value[]} ~> $each(function($v, $k) {
{
"locale": $k,
"metadata": $v.{"key": ???,"value": $}
}
})
}
Please help me to fill "???" so that I can get the array name
Assuming that the input object will always have a single root-level key you can write your expression like this:
{
"data": title{locale: value[]} ~> $each(function($v, $k) {
{
"locale": $k,
"metadata": $v.{"key": $keys($$)[0],"value": $}
}
})
}
$keys returns an array containing keys in the object. $keys($$) will return all keys in root-level of this array (in this case: "title").
Note that for a following input object:
{"title": [
{
"value": "18724-100",
"locale": "en-GB"
},
{
"value": "18724-5",
"locale": "en-GB"
},
{
"value": "18724-99",
"locale": "fr-FR"
}
],
"foo": 123
}
$keys($$) would return an array of two elements (["title", "foo"]).

scala ujson.read() returns ujson.Obj

I am trying to read a json string using Li Haoyi's ujson. This is the string:
{
"dataflows": [
{
"name": "test",
"sources": [
{
"name": "person_inputs",
"path": "/data/input/events/person/*",
"format": "JSON"
}
],
"transformations": [
{
"name": "validation",
"type": "validate_fields",
"params": {
"input": "person_inputs",
"validations": [
{
"field": "office",
"validations": [
"notEmpty"
]
},
{
"field": "age",
"validations": [
"notNull"
]
}
]
}
},
{
"name": "ok_with_date",
"type": "add_fields",
"params": {
"input": "validation_ok",
"addFields": [
{
"name": "dt",
"function": "current_timestamp"
}
]
}
}
],
"sinks": [
{
"input": "ok_with_date",
"name": "raw-ok",
"paths": [
"/data/output/events/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
},
{
"input": "validation_ko",
"name": "raw-ko",
"paths": [
"/data/output/discards/person"
],
"format": "JSON",
"saveMode": "OVERWRITE"
}
]
}
]
}
And this is how I read it:
val j = os.read(os.pwd/RelPath("src/main/scala/metadata.json"))
val jsonData = ujson.read(j)
But, the return type is ujson.Obj, and not Arr(ArrayBuffer(Obj), as expected, such that when I try to get jsonData(0), what I get is json.Value$InvalidData: Expected ujson.Arr.
I am asking this question because I have tried to use the ujson object to create a upickle object, but I cannot, and I suspect it is because of this initial error.
Any ideas of why this happens? Any help would be greatly appreciated! Thanks in advance!!
The outer element of your JSON is not an array, it is an object with a single element dataflows whose value is an array. Try jsonData("dataflows")(0).

How to filter json file with specific tag values and get output into csv

I have created a json file with the output having key values pair. But I would like to filter more and get only specific tags and get new output in table using excel (csv) format
aws resourcegroupstaggingapi get-resources --tags-per-page 100 --tag-filters Key=ProjectName,Values=Avengers > tag-filter.json
However it provides the list of all the tags besides "ProjectName". I would like to filter the output with 2 more tags with their values but not all of them:
Actual results:
{
"ResourceTagMappingList": [
{
"ResourceARN": "arn:aws:app:us-east-1:XXXX/mesh/Avenger1",
"Tags": [
{
"Key": "ApplicationName",
"Value": "HULK"
},
{
"Key": "Owner",
"Value": "Mark Ruffalo"
},
{
"Key": "Costume",
"Value": "GREEN"
},
{
"Key": "Power",
"Value": "SMASH"
},
{
"Key": "ProjectName",
"Value": "Avengers"
}
]
},
{
"ResourceARN": "arn:aws:app:us-east-1:XXXX:mesh/Avenger2",
"Tags": [
{
"Key": "ApplicationName",
"Value": "IRON-MAN"
},
{
"Key": "Owner",
"Value": "Robert Downey Jr."
},
{
"Key": "Costume",
"Value": "RED"
},
{
"Key": "Power",
"Value": "SuperSonic"
},
{
"Key": "ProjectName",
"Value": "Avengers"
}
]
}
]
}
Expected Results:
{
"ResourceTagMappingList": [
{
"ResourceARN": "arn:aws:app:us-east-1:XXXX/mesh/Avenger1",
"Tags": [
{
"Key": "ApplicationName",
"Value": "HULK"
},
{
"Key": "Owner",
"Value": "Mark Ruffalo"
},
{
"Key": "ProjectName",
"Value": "Avengers"
}
]
},
{
"ResourceARN": "arn:aws:app:us-east-1:XXXX:mesh/Avenger2",
"Tags": [
{
"Key": "ApplicationName",
"Value": "IRON-MAN"
},
{
"Key": "Owner",
"Value": "Robert Downey Jr."
},
{
"Key": "ProjectName",
"Value": "Avengers"
}
]
}
]
}
To achieve the "expected" output given the "actual" output, you could use the following filter:
.ResourceTagMappingList[].Tags
|= map(select(.Key|IN("ApplicationName","Owner","ProjectName")))
To achieve the expected CSV, it would be helpful to know what you expect.

Parse JSON with map of list

I am new to scala and JSON parsing and need some help. I need to parse the complex JSON (below) to get the values of "name" in "dimension" key i.e I need PLATFORM and OS_VERSION.
I tried multiple options, but it is not working. Any help is appreciated
This is a snippet of the code I tried, but I am not able to proceed further in parsing the list. I believe the 'ANY' keyword is causing some mismatch / issues.
import org.json4s._
import org.json4s.jackson.JsonMethods._
implicit val formats = org.json4s.DefaultFormats
val mapJSON = parse(tmp).extract[Map[String, Any]]
println(mapJSON)
//for ((k,v) <- mapJSON) printf("key: %s, value: %s\n", k, v)
val list_map = mapJSON("dimensions")
{
"uuid": "uuidddd",
"last_modified": 1559080222953,
"version": "2.6.1.0",
"name": "FULL_DAY_2_mand_date",
"is_draft": false,
"model_name": "FULL_DAY_1_may05",
"description": "",
"null_string": null,
"dimensions": [
{
"name": "PLATFORM",
"table": "tbl1",
"column": "PLATFORM",
"derived": null
},
{
"name": "OS_VERSION",
"table": "tbl1",
"column": "OS_VERSION",
"derived": null
},
],
"measures": [
{
"name": "_COUNT_",
"function": {
"expression": "COUNT",
"parameter": {
"type": "constant",
"value": "1"
},
"returntype": "bigint"
}
},
{
"name": "UU",
"function": {
"expression": "COUNT_DISTINCT",
"parameter": {
"type": "column",
"value": "tbl1.USER_ID"
},
"returntype": "hllc(12)"
}
},
{
"name": "CONT_SIZE",
"function": {
"expression": "SUM",
"parameter": {
"type": "column",
"value": "tbl1.SIZE"
},
"returntype": "bigint"
}
},
{
"name": "CONT_COUNT",
"function": {
"expression": "SUM",
"parameter": {
"type": "column",
"value": "tbl1.COUNT"
},
"returntype": "bigint"
}
}
],
"dictionaries": [],
"rowkey": {
"rowkey_columns": [
{
"column": "tbl1.OS_VERSION",
"encoding": "dict",
"encoding_version": 1,
"isShardBy": false
},
{
"column": "tbl1.PLATFORM",
"encoding": "dict",
"encoding_version": 1,
"isShardBy": false
},
{
"column": "tbl1.DEVICE_FAMILY",
"encoding": "dict",
"encoding_version": 1,
"isShardBy": false
}
]
},
"hbase_mapping": {
"column_family": [
{
"name": "F1",
"columns": [
{
"qualifier": "M",
"measure_refs": [
"_COUNT_",
"CONT_SIZE",
"CONT_COUNT"
]
}
]
},
{
"name": "F2",
"columns": [
{
"qualifier": "M",
"measure_refs": [
"UU"
]
}
]
}
]
},
"aggregation_groups": [
{
"includes": [
"tbl1.PLATFORM",
"tbl1.OS_VERSION"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [
"tbl1.DATE_HR"
],
"joint_dims": []
}
}
],
"signature": "ttrrs==",
"notify_list": [],
"status_need_notify": [
"ERROR",
"DISCARDED",
"SUCCEED"
],
"partition_date_start": 0,
"partition_date_end": 3153600000000,
"auto_merge_time_ranges": [
604800000,
2419200000
],
"volatile_range": 0,
"retention_range": 0,
"engine_type": 4,
"storage_type": 2,
"override_kylin_properties": {
"job.queuename": "root.production.P0",
"is-mandatory-only-valid": "true"
},
"cuboid_black_list": [],
"parent_forward": 3,
"mandatory_dimension_set_list": [],
"snapshot_table_desc_list": []
}
You need to make more specific classes for parsing the data, something like this:
case class Dimension(name: String, table: String, column: String)
case class AllData(uuid: String, dimensions: List[Dimension])
val data = parse(tmp).extract[AllData]
val names = data.dimensions.map(_.name)