I am trying to fetch data from nested JSON, I need only a few fields from the JSON,
I have created case classes for the required data, the solution I found from google suggested to use read function, but I get an empty Object
I tried to google with no success, What I am missing?
my code
val rawDataFromFile = Source.fromFile(path).mkString
case class Data(listOfPersons: List[Person])
case class Person(bio: Bio, terms: List[Term])
case class Bio(birthday: String, gender: String)
case class Term(`type`: String, start: String, end: String)
read[Data](rawDataFromFile)
res >> Data(List())
and the JSON
[
{
"id": {
"not_intresting_field_1": "B000944",
"not_intresting_field_4": [
"H2OH13033",
"S6OH00163"
]
},
"name": {
"first": "first_name_1",
"last": "last_name_1"
},
"bio": {
"birthday": "1952-11-09",
"gender": "M"
},
"terms": [
{
"type": "rep",
"start": "1993-01-05",
"end": "1995-01-03"
},
{
"type": "rep",
"start": "1995-01-04",
"end": "1997-01-03"
}
]
},
{
"id": {
"not_intresting_field_1": "C000127",
"not_intresting_field_4": [
"S8WA00194",
"H2WA01054"
]
},
"name": {
"first": "first_name_1",
"last": "last_name_1"
},
"bio": {
"birthday": "1958-10-13",
"gender": "F"
},
"terms": [
{
"type": "rep",
"start": "1993-01-05",
"end": "1995-01-03"
},
{
"type": "sen",
"start": "2001-01-03",
"end": "2007-01-03"
}
]
}
]
Your case class is not the same as your json structure.
Here your define Data type which will read json like following
{
"listOfPersons": [
{
"id": {
"not_intresting_field_1": "B000944",
"not_intresting_field_4": [
"H2OH13033",
"S6OH00163"
]
},
"name": {
"first": "first_name_1",
"last": "last_name_1"
},
"bio": {
"birthday": "1952-11-09",
"gender": "M"
},
... //your original json
}
]
}
Try this
read[List[Person]](rawDataFromFile)
Related
Lets say I have the following JSON structure in my DB:
{
"user": "test",
"cars": [
{
"name": "car1",
"drivers": [
{
"name": "name1",
"age": 18
},
{
"name": "name2",
"age": 21
}
]
},
{
"name": "car2",
"drivers": [
{
"name": "name3",
"age": 25
}
]
}
]
}
I want to remove all the age fields in the nested arrays?
How can I achieve this via MySQL JSON functions?
using couchbase 5
I need to build a query string from this object
[
{
"_id": 190,
"querystring": [
{
"name": "p1",
"value": "val1"
},
{
"name": "p2",
"value": "val2"
}
]
}
]
the expected output should be
p1=val1&p2=val2
can anyone help here?
after few attempts I think I got closer to the solution I need.
[
{
"_id": 190,
"res": [
"company_id=$PREFIJO&",
"user_country=$COUNTRY&",
"offer_unique_code=$PIXEL&",
"pub_id=$PUBID&"
]
}
]
now, how can I convert "res" to a concatenated string of all the array elements?
WITH obj AS ({ "_id": 190, "querystring": [ { "name": "p1", "value": "val1" }, { "name": "p2", "value": "val2" } ] })
SELECT obj._id, CONCAT2("&", ARRAY CONCAT2("=",v.name,v.`value`) FOR v IN obj.querystring END) AS res;
Array of objects
WITH objs AS ([{ "_id": 190, "querystring": [ { "name": "p1", "value": "val1" }, { "name": "p2", "value": "val2" } ] },
{ "_id": 191, "querystring": [ { "name": "p3", "value": "val1" }, { "name": "p4", "value": "val2" } ] }
])
SELECT obj._id, CONCAT2("&", ARRAY CONCAT2("=",v.name,v.`value`) FOR v IN obj.querystring END) AS res FROM objs AS obj ;
Older version where CONCAT2() not available, get array of strings (name=val) and do in application or use the following technique. Assume your name/val doesn't have any replace characters.
WITH objs AS ([{ "_id": 190, "querystring": [ { "name": "p1", "value": "val1" }, { "name": "p2", "value": "val2" } ] },
{ "_id": 191, "querystring": [ { "name": "p3", "value": "val1" }, { "name": "p4", "value": "val2" } ] }
])
SELECT obj._id, replace(replace(replace(encode_json(ARRAY CONCAT(v.name,"=",v.`value`) FOR v IN obj.querystring END),"\",\"","&"),"[\"",""),"\"]","") AS res FROM objs AS obj ;
If single document then have ARRAY of objects then use UNNEST
If there is number , convert to string using TO_STR() before CONCAT operation
https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/stringfun.html#fn-str-concat2
I have an array of json objects like this
[
{
"events": [
{
"type": "message",
"attributes": [
{
"key": "action",
"value": "withdraw_reward"
},
{
"key": "sender",
"value": "bob"
},
{
"key": "module",
"value": "distribution"
},
{
"key": "sender",
"value": "bob"
}
]
},
{
"type": "credit",
"attributes": [
{
"key": "recipient",
"value": "ross"
},
{
"key": "sender",
"value": "bob"
},
{
"key": "amount",
"value": "100"
}
]
},
{
"type": "rewards",
"attributes": [
{
"key": "amount",
"value": "100"
},
{
"key": "validator",
"value": "sarah"
}
]
}
]
},
{
"events": [
{
"type": "message",
"attributes": [
{
"key": "action",
"value": "withdraw_reward"
},
{
"key": "sender",
"value": "bob"
},
{
"key": "module",
"value": "distribution"
},
{
"key": "sender",
"value": "bob"
}
]
},
{
"type": "credit",
"attributes": [
{
"key": "recipient",
"value": "ross"
},
{
"key": "sender",
"value": "bob"
},
{
"key": "amount",
"value": "100"
}
]
},
{
"type": "rewards",
"attributes": [
{
"key": "amount",
"value": "200"
},
{
"key": "validator",
"value": "Ryan"
}
]
}
]
}
]
How to traverse through the types, check if it's type equals to rewards and then go through the attributes and verify if the validator equals to sarah and fetch the value of the key amount? Pretty new to scala and play framework. Any help would be great. Thanks
You could parse your JSON into a structure of case classes for easier handling and then extract the wanted field like so:
val json =
"""[
{"events":[
{
"type":"message","attributes":[
{"key":"action","value":"withdraw_reward"},
{"key":"sender","value":"bob"},
{"key":"module","value":"distribution"},
{"key":"sender","value":"bob"}
]},
{
"type":"credit","attributes":[
{"key":"recipient","value":"ross"},
{"key":"sender","value":"bob"},
{"key":"amount","value":"100"}
]},
{
"type":"rewards","attributes":[
{"key":"amount","value":"100"},
{"key":"validator","value":"sara"}
]}
]
},
{"events":[
{
"type":"message","attributes":[
{"key":"action","value":"withdraw_reward"},
{"key":"sender","value":"bob"},
{"key":"module","value":"distribution"},
{"key":"sender","value":"bob"}
]},
{
"type":"credit","attributes":[
{"key":"recipient","value":"ross"},
{"key":"sender","value":"bob"},
{"key":"amount","value":"100"}
]},
{
"type":"rewards","attributes":[
{"key":"amount","value":"200"},
{"key":"validator","value":"Ryan"}
]}
]
}
]
"""
case class EventWrapper(events: Seq[Event])
case class KeyValue(key: String, value: String)
case class Event(`type`: String, attributes: Seq[KeyValue])
import play.api.libs.json._
implicit val kvReads: Reads[KeyValue] = Json.reads[KeyValue]
implicit val eventReads: Reads[Event] = Json.reads[Event]
implicit val eventWrapperReads: Reads[EventWrapper] = Json.reads[EventWrapper]
val rewardAmountsValidatedBySara = Json
.parse(json)
.as[Seq[EventWrapper]]
.flatMap {
_.events.collect {
case Event(t, attributes) if t == "rewards" && attributes.contains(KeyValue("validator", "sara")) =>
attributes.collect {
case KeyValue("amount", value) => value
}
}.flatten
}
val amount = rewardAmountsValidatedBySara.head
For your example, rewardAmountsValidatedBySara would yield a List of Strings containing only the String "100". Which you could retrieve (potentially unsafe) with .head as shown above.
Normally you would not do this, as it could throw an exception on an empty List, so it would be better to use .headOption which returns an Option which you can then handle safely.
Note that the implicit Reads are Macros, which automatically translate into Code, that instructs the Play Json Framework how to read the JsValue into the defined case classes, see the documentation for more info.
In the Json string given below, I want to find all elements in which category = m AND the "middle" array contains elements which match this condition - the element's "middle" array has objects whose itemType = Executable.
I would like to use jsonpath to get the desired objects. I prefer to not use jmespath because it can be too complex for my purpose. But, I am new to jsonpath and I am not able to figure out the json query from online tutorials which are too trivial or basic. I wonder if its better to use a programming language instead to get the data I need. Please advise.
So far, I was able to only extract elements in which category = m by using this jsonpath query $.[?(#.category=="m")]. How do I do the remaining part ?
Json :
Overview - Every object has a "content" object. Each content object generally has a start, middle and end array besides other fields. Middle arrays can have multiple content objects inside them and so on. Some of the content objects have only a middle array. I am interested in locating items in such content objects as mentioned above.
Note that this is not the actual json which I have to process. It is an imitation which has been sanitized for SO.
{
"id": "123",
"contents": {
"title": "B1",
"start": [],
"middle": [
{
"level": "1",
"contents": {
"title": "C1",
"category": "c",
"start": [],
"middle": [
{
"level": "2",
"contents": {
"title": "M1",
"category": "m",
"start": [],
"middle": [
{
"level": "3",
"contents": {
"title": "MAT1",
"middle": [
{
"itemType": "Data"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT2",
"middle": [
{
"itemType": "Executable",
"id": "exec1"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT3",
"middle": [
{
"itemType": "Data"
}
]
}
}
],
"end": []
}
},
{
"level": "2",
"contents": {
"title": "M2",
"category": "m",
"start": [],
"middle": [
{
"level": "3",
"contents": {
"title": "MAT1",
"middle": [
{
"itemType": "Data"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT2",
"middle": [
{
"itemType": "Executable",
"id": "exec2"
}
]
}
}
],
"end": []
}
}
],
"end": []
}
},
{
"level": "1",
"contents": {
"title": "C2",
"category": "c",
"start": [],
"middle": [
{
"level": "2",
"contents": {
"title": "M1",
"category": "m",
"start": [],
"middle": [
{
"level": "3",
"contents": {
"title": "MAT1",
"middle": [
{
"itemType": "Data"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT2",
"middle": [
{
"itemType": "Executable",
"id": "exec3"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT3",
"middle": [
{
"itemType": "Data"
}
]
}
}
],
"end": []
}
},
{
"level": "2",
"contents": {
"title": "M2",
"category": "m",
"start": [],
"middle": [
{
"level": "3",
"contents": {
"title": "MAT1",
"middle": [
{
"itemType": "Data"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT2",
"middle": [
{
"itemType": "Executable",
"id": "exec4"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT3",
"middle": [
{
"itemType": "Data"
}
]
}
}
],
"end": []
}
}
],
"end": []
}
}
],
"end": []
}
}
Context
json with nested objects1
jsonpath expression language
choosing between jsonpath and jmespath (or other JSON expression engine)
Problem
DeveMasterJoe2 wants to extract some values from nested JSON
Discussion
There are lots of implementations of jsonpath out there, and they do not all support the same features
The structure and normalization of the source JSON is going to influence how easily this can be done with pure jsonpath
In choosing a JSON expression engine, one has to weigh multiple factors
how consistent are the implementations across languages?
how many choices are there within a given language?
how clear is the specification?
how many examples, unit-tests or tutorials are available?
who is supporting it?
Example solution using Python and jsonpath-ng
Here is an example solution using python 3.7 and jsonpath-ng
This example uses a mix of jsonpath and python instead of just pure jsonpath, because of the heavily-nested JSON
I will leave it for someone else to provide an answer that relies on pure jsonpath
Note that the source JSON arguably could stand to be cleaned up a bit
(for example, why is there no id field attached to itemType==Data elements?)
(for example, why is category not found on all contents elements?)
(for example, if you expressly specify level why complicate things with heavily nested objects when you can determine depth by level ?)
This example:
## import libraries
import codecs
import json
import jsonpath_ng
from jsonpath_ng.ext import parse
##;;
## init vars
href="path/to/my/jsonfile/nested_dict.json"
json_string = codecs.open(href, 'rb', encoding='utf8').read()
json_dataroot = json.loads(json_string)
final_result = []
##;;
## init jsonpath outer-query
match = parse('$..contents.middle[*]').find(json_dataroot)
##;;
## iterate through outer-query and gather subelements
for ijj,item in enumerate(match):
## restrict to desired category == 'm'
if(match[ijj].value.get('contents',{}).get('category','') == 'm'):
## extract out desired subelements
json_datafrag001 = [item.get('contents',{}).get('middle',{})[0]
for item in match[ijj].value.get('contents',{}).get('middle',{})
]
match001 = parse("$[?(#.itemType=='Executable')]").find(json_datafrag001)
final_result.extend(list(match001[ikk].value for ikk,item in enumerate(match001)))
pass
##;;
## show final result
vout = json.dumps(final_result, sort_keys=True,indent=4, separators=(',', ': '))
print(vout)
##;;
... produces this result ...
[
{
"id": "exec1",
"itemType": "Executable"
},
{
"id": "exec2",
"itemType": "Executable"
},
{
"id": "exec3",
"itemType": "Executable"
},
{
"id": "exec4",
"itemType": "Executable"
}
]
1 (aka dictionary, associative-array, hash)
I'm new to Groovy. But, I can try with some simple code lines to extract the data.
I have read the from Parse JSON using groovy script (using JsonSlurper). But it didn't help in my case.
My code line:
{
"errors": false,
"address_data": [
{
"address_id": "567",
"township": {
"id": "41079",
"name": "Test Data"
},
"city": {
"id": "1622",
"name": "Test City"
},
"region": {
"id": "663",
"name": "Metro Test"
},
"stock_source_code": "Test",
"is_default_address": false
},
{
"address_id": "45444",
"township": {
"id": "41079",
"name": "Test Test"
},
"city": {
"id": "1622",
"name": "Test City"
},
"region": {
"id": "663",
"name": "Metro Test Taguig"
},
"is_default_address": true
},
{
"address_id": "45444",
"township": {
"id": "888888",
"name": "Apas"
},
"city": {
"id": "432",
"name": "Test City"
},
"region": {
"id": "591",
"name": "Test Cebu"
},
"stock_source_code": "testce",
"is_default_address": false
}
]
}
My code lines:
def response = "data_above";
def object = new JsonSlurper().parseText(response);
def errors = object.errors
if (errors == false) { //could access object.errors property
log.info "Checking condition"
def addressData = object.address_data // Cannot get the data
}
How can I get the address_data property?