I'm using JSONPath in AWS Step Functions (playground: Parameters tab).
In this example if we have a JSON input:
{
"movies": [
{
"genre": "crime",
"director": "Quentin Tarantino",
"title": "Reservoir Dogs",
"year": 1992
},
{
"genre": "action",
"director": "Brian De Palma",
"title": "Mission: Impossible",
"year": 1996,
"staring": [
"Tom Cruise"
]
}
],
"metadata": {
"lastUpdated": "2020-05-27T08:00:00.000Z"
},
"stringJson": "{\"arr\": [1, 2, 3, 4, 5], \"bool\": true, \"null\": null, \"number\": 1}"
}
and parameters like this:
{
"test.$": "$.metadata"
}
we would get this output:
{
"test": {
"lastUpdated": "2020-05-27T08:00:00.000Z"
}
}
while I would like to get
{
"lastUpdated": "2020-05-27T08:00:00.000Z"
}
or even better, say all but one:
{
"movies": [
{
"genre": "crime",
"director": "Quentin Tarantino",
"title": "Reservoir Dogs",
"year": 1992
},
{
"genre": "action",
"director": "Brian De Palma",
"title": "Mission: Impossible",
"year": 1996,
"staring": [
"Tom Cruise"
]
}
],
"metadata": {
"lastUpdated": "2020-05-27T08:00:00.000Z"
}
}
How do I put an element (or all elements) in the root?
I tried parameters like:
{
"$": "$.metadata.lastUpdated"
}
or
{
"$": "$.(?[# != 'stringJson'])"
}
but none of these worked for me.
Set InputPath to $.metadata to get:
{
"lastUpdated": "2020-05-27T08:00:00.000Z"
}
Reference paths (the type of path that InputPath accepts) is limited in that it can only identify a single node within the entire JSON structure.
You can't natively exclude all elements except 'x' element.
Related
In the Json string given below, I want to find all elements in which category = m AND the "middle" array contains elements which match this condition - the element's "middle" array has objects whose itemType = Executable.
I would like to use jsonpath to get the desired objects. I prefer to not use jmespath because it can be too complex for my purpose. But, I am new to jsonpath and I am not able to figure out the json query from online tutorials which are too trivial or basic. I wonder if its better to use a programming language instead to get the data I need. Please advise.
So far, I was able to only extract elements in which category = m by using this jsonpath query $.[?(#.category=="m")]. How do I do the remaining part ?
Json :
Overview - Every object has a "content" object. Each content object generally has a start, middle and end array besides other fields. Middle arrays can have multiple content objects inside them and so on. Some of the content objects have only a middle array. I am interested in locating items in such content objects as mentioned above.
Note that this is not the actual json which I have to process. It is an imitation which has been sanitized for SO.
{
"id": "123",
"contents": {
"title": "B1",
"start": [],
"middle": [
{
"level": "1",
"contents": {
"title": "C1",
"category": "c",
"start": [],
"middle": [
{
"level": "2",
"contents": {
"title": "M1",
"category": "m",
"start": [],
"middle": [
{
"level": "3",
"contents": {
"title": "MAT1",
"middle": [
{
"itemType": "Data"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT2",
"middle": [
{
"itemType": "Executable",
"id": "exec1"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT3",
"middle": [
{
"itemType": "Data"
}
]
}
}
],
"end": []
}
},
{
"level": "2",
"contents": {
"title": "M2",
"category": "m",
"start": [],
"middle": [
{
"level": "3",
"contents": {
"title": "MAT1",
"middle": [
{
"itemType": "Data"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT2",
"middle": [
{
"itemType": "Executable",
"id": "exec2"
}
]
}
}
],
"end": []
}
}
],
"end": []
}
},
{
"level": "1",
"contents": {
"title": "C2",
"category": "c",
"start": [],
"middle": [
{
"level": "2",
"contents": {
"title": "M1",
"category": "m",
"start": [],
"middle": [
{
"level": "3",
"contents": {
"title": "MAT1",
"middle": [
{
"itemType": "Data"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT2",
"middle": [
{
"itemType": "Executable",
"id": "exec3"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT3",
"middle": [
{
"itemType": "Data"
}
]
}
}
],
"end": []
}
},
{
"level": "2",
"contents": {
"title": "M2",
"category": "m",
"start": [],
"middle": [
{
"level": "3",
"contents": {
"title": "MAT1",
"middle": [
{
"itemType": "Data"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT2",
"middle": [
{
"itemType": "Executable",
"id": "exec4"
}
]
}
},
{
"level": "3",
"contents": {
"title": "MAT3",
"middle": [
{
"itemType": "Data"
}
]
}
}
],
"end": []
}
}
],
"end": []
}
}
],
"end": []
}
}
Context
json with nested objects1
jsonpath expression language
choosing between jsonpath and jmespath (or other JSON expression engine)
Problem
DeveMasterJoe2 wants to extract some values from nested JSON
Discussion
There are lots of implementations of jsonpath out there, and they do not all support the same features
The structure and normalization of the source JSON is going to influence how easily this can be done with pure jsonpath
In choosing a JSON expression engine, one has to weigh multiple factors
how consistent are the implementations across languages?
how many choices are there within a given language?
how clear is the specification?
how many examples, unit-tests or tutorials are available?
who is supporting it?
Example solution using Python and jsonpath-ng
Here is an example solution using python 3.7 and jsonpath-ng
This example uses a mix of jsonpath and python instead of just pure jsonpath, because of the heavily-nested JSON
I will leave it for someone else to provide an answer that relies on pure jsonpath
Note that the source JSON arguably could stand to be cleaned up a bit
(for example, why is there no id field attached to itemType==Data elements?)
(for example, why is category not found on all contents elements?)
(for example, if you expressly specify level why complicate things with heavily nested objects when you can determine depth by level ?)
This example:
## import libraries
import codecs
import json
import jsonpath_ng
from jsonpath_ng.ext import parse
##;;
## init vars
href="path/to/my/jsonfile/nested_dict.json"
json_string = codecs.open(href, 'rb', encoding='utf8').read()
json_dataroot = json.loads(json_string)
final_result = []
##;;
## init jsonpath outer-query
match = parse('$..contents.middle[*]').find(json_dataroot)
##;;
## iterate through outer-query and gather subelements
for ijj,item in enumerate(match):
## restrict to desired category == 'm'
if(match[ijj].value.get('contents',{}).get('category','') == 'm'):
## extract out desired subelements
json_datafrag001 = [item.get('contents',{}).get('middle',{})[0]
for item in match[ijj].value.get('contents',{}).get('middle',{})
]
match001 = parse("$[?(#.itemType=='Executable')]").find(json_datafrag001)
final_result.extend(list(match001[ikk].value for ikk,item in enumerate(match001)))
pass
##;;
## show final result
vout = json.dumps(final_result, sort_keys=True,indent=4, separators=(',', ': '))
print(vout)
##;;
... produces this result ...
[
{
"id": "exec1",
"itemType": "Executable"
},
{
"id": "exec2",
"itemType": "Executable"
},
{
"id": "exec3",
"itemType": "Executable"
},
{
"id": "exec4",
"itemType": "Executable"
}
]
1 (aka dictionary, associative-array, hash)
I have an array of JSON objects formatted as follows:
[
{
"id": 1,
"names": [
{
"name": "Bulbasaur",
"language": {
"name": "en",
"url": "http://myserver.com:8000/api/v2/language/9/"
}
},
],
},
{
"id": 1,
"types": [
{
"slot": 1,
"type": {
"name": "grass",
"url": "http://myserver.com:8000/api/v2/type/12/"
}
},
{
"slot": 2,
"type": {
"name": "poison",
"url": "http://myserver.com:8000/api/v2/type/4/"
}
}
]
},
{
"id": 2,
"names": [
{
"name": "Ivysaur",
"language": {
"name": "en",
"url": "http://myserver.com:8000/api/v2/language/9/"
}
},
],
},
{
"id": 2,
"types": [
{
"slot": 1,
"type": {
"name": "ice",
"url": "http://myserver.com:8000/api/v2/type/10/"
}
},
{
"slot": 2,
"type": {
"name": "electric",
"url": "http://myserver.com:8000/api/v2/type/8/"
}
}
]
},
{
"id": 3,
"names": [
{
"name": "Venusaur",
"language": {
"name": "en",
"url": "http://myserver.com:8000/api/v2/language/9/"
}
},
],
},
{
"id": 3,
"types": [
{
"slot": 1,
"type": {
"name": "ground",
"url": "http://myserver.com:8000/api/v2/type/2/"
}
},
{
"slot": 2,
"type": {
"name": "rock",
"url": "http://myserver.com:8000/api/v2/type/3/"
}
}
]
}
]
Note that these are pairs of separate objects that appear sequentially in a JSON array, with each pair sharing an id field. This pattern repeats several hundred times in the array. What I need to accomplish is to "merge" each id-sharing pair into one object. So, the resultant output would be
[
{
"id": 1,
"names": [
{
"name": "Bulbasaur",
"language": {
"name": "en",
"url": "http://myserver.com:8000/api/v2/language/9/"
}
},
],
"types": [
{
"slot": 1,
"type": {
"name": "grass",
"url": "http://myserver.com:8000/api/v2/type/12/"
}
},
{
"slot": 2,
"type": {
"name": "poison",
"url": "http://myserver.com:8000/api/v2/type/4/"
}
}
]
},
{
"id": 2,
"names": [
{
"name": "Ivysaur",
"language": {
"name": "en",
"url": "http://myserver.com:8000/api/v2/language/9/"
}
},
],
"types": [
{
"slot": 1,
"type": {
"name": "ice",
"url": "http://myserver.com:8000/api/v2/type/10/"
}
},
{
"slot": 2,
"type": {
"name": "electric",
"url": "http://myserver.com:8000/api/v2/type/8/"
}
}
]
},
{
"id": 3,
"names": [
{
"name": "Venusaur",
"language": {
"name": "en",
"url": "http://myserver.com:8000/api/v2/language/9/"
}
},
],
"types": [
{
"slot": 1,
"type": {
"name": "ground",
"url": "http://myserver.com:8000/api/v2/type/2/"
}
},
{
"slot": 2,
"type": {
"name": "rock",
"url": "http://myserver.com:8000/api/v2/type/3/"
}
}
]
}
]
I've gotten these objects to appear next to each other via the group_by(.id) command, but I'm at a loss as to how I should actually combine them. I'm very much still a novice with jq so I'm a bit overwhelmed with the amount of possible solutions.
[Note: The following assumes that the data shown in the Q have been corrected so that they are valid JSON.]
The merging you want can be achieved by object addition (x + y). For example, given the two JSON objects as shown in the question (i.e., as a stream), you could write:
jq -s '.[0] + .[1]'
However, since the question also indicates these objects are actually in an array, let's next consider the case of an array with two objects. In that case, you could simply write:
jq add
Finally, if you have an array of arrays each of which is an array of objects, you could use map(add). Since you don't have a very large array, you could simply write:
group_by(.id) | map(add)
Please note that jq defines object addition in a non-commutative way. Specifically, there is a bias towards the right-most key.
I have a json file and I want to add some value from top in another place in json.
I am trying to use jq command line.
{
"channel": "mychannel",
"videos": [
{
"id": "10",
"url": "youtube.com"
},
{
"id": "20",
"url": "youtube.com"
}
]
}
The output would be:
{
"channel": "mychannel",
"videos": [
{
"channel": "mychannel",
"id": "10",
"url": "youtube.com"
},
{
"channel": "mychannel",
"id": "20",
"url": "youtube.com"
}
]
}
in my json the "channel" is static, same value always. I need a way to concatenate always in each video array.
Someone can help me?
jq .videos + channel
Use a variable to remember .channel in the later stages of the pipeline.
$ jq '.channel as $ch | .videos[].channel = $ch' tmp.json
{
"channel": "mychannel",
"videos": [
{
"id": "10",
"url": "youtube.com",
"channel": "mychannel"
},
{
"id": "20",
"url": "youtube.com",
"channel": "mychannel"
}
]
}
I'm trying to write extract the name value "Acura" from a JSON array response by using the location of the value stored in a variable called "jsonFieldName".
Below is the code that I'm trying to do this with, however, everytime i run the script, SOAPUI returns error: "java.lang.NullPointerException: Cannot get property 'name' on null object error at line: 156"
Can someone kindly advise how to do this?
import groovy.json.JsonSlurper
def response = '''{
"makes": [
{
"id": 200002038,
"name": "Acura",
"niceName": "acura",
"models": [
{
"id": "Acura_ILX",
"name": "ILX",
"niceName": "ilx",
"years": [
{
"id": 200471908,
"year": 2014
}
]
},
{
"id": "Acura_ILX_Hybrid",
"name": "ILX Hybrid",
"niceName": "ilx-hybrid",
"years": [
{
"id": 200493809,
"year": 2014
}
]
},
{
"id": "Acura_MDX",
"name": "MDX",
"niceName": "mdx",
"years": [
{
"id": 200465929,
"year": 2014
}
]
},
{
"id": "Acura_RDX",
"name": "RDX",
"niceName": "rdx",
"years": [
{
"id": 200467168,
"year": 2014
}
]
},
{
"id": "Acura_RLX",
"name": "RLX",
"niceName": "rlx",
"years": [
{
"id": 100539511,
"year": 2014
}
]
},
{
"id": "Acura_TL",
"name": "TL",
"niceName": "tl",
"years": [
{
"id": 200488448,
"year": 2014
}
]
},
{
"id": "Acura_TSX",
"name": "TSX",
"niceName": "tsx",
"years": [
{
"id": 200490517,
"year": 2014
}
]
},
{
"id": "Acura_TSX_Sport_Wagon",
"name": "TSX Sport Wagon",
"niceName": "tsx-sport-wagon",
"years": [
{
"id": 200673755,
"year": 2014
}
]
}
]
},
{
"id": 200001769,
"name": "Aston Martin",
"niceName": "aston-martin",
"models": [
{
"id": "Aston_Martin_DB9",
"name": "DB9",
"niceName": "db9",
"years": [
{
"id": 200473436,
"year": 2014
}
]
},
{
"id": "Aston_Martin_Rapide_S",
"name": "Rapide S",
"niceName": "rapide-s",
"years": [
{
"id": 200460643,
"year": 2014
}
]
},
{
"id": "Aston_Martin_V8_Vantage",
"name": "V8 Vantage",
"niceName": "v8-vantage",
"years": [
{
"id": 200472947,
"year": 2014
}
]
},
{
"id": "Aston_Martin_Vanquish",
"name": "Vanquish",
"niceName": "vanquish",
"years": [
{
"id": 200431313,
"year": 2014
}
]
}
]
}
],
"makesCount": 2
}'''
def jsonFieldName = ('makes[0].name')
def json = new JsonSlurper().parseText (response)
jsonFieldName.split("\\.").each{json = json[it]}
assert json == 'Acura'
Assuming that your JSON is good from the response (check by calling print) Try adding .text to the end of your jsonSlurper() Call
It also looks like you have a space between parseText and (response)
so it should be
def json = new JsonSlurper().parseText(response)
However I would try casting to ArrayList<LazyMap> to ensure you can iterate by
ArrayList<LazyMap> json = new JsonSlurper().parseText(response) as ArrayList<LazyMap>
Then call :
json.get('Acura')
This line of your code doesn't handle the index resolution:
jsonFieldName.split("\\.").each{json = json[it]}
There is no key with the name makes[0]. Instead there is an array of makes and you are interested on the first one. The following hardcoded line retrieves the name attribute:
def result = json.'makes'[0].'name'
As you can see here there is an additional step to resolve the index operator. Of course you can implement this functionality on your own or use JsonPath instead of JsonSlurper.
OK so I managed to get this working by using JsonPath instead of JsonSlurper.
To achieve this I had to import the following:
import com.jayway.jsonpath.JsonPath
def jsonFieldName = "makes[0].name"
def expectedValue = "Acura"
def jsonSuff = JsonPath.read(response, jsonFieldName)
log.info(jsonSuff)
if (jsonSuff.toString() == expectedValue.toString()){
log.info("Actual value"+jsonSuff+"is equal to expected value"+expectedValue)
}
I have a JSON document like
{
"branch": [
{
"section": [
{
"sub": "edc",
"time": "one hour",
"frequency": "3"
},
{
"sub": "bee",
"time": "two hours",
"frequency": "4"
}
]
},
{
"section": [
{
"sub": "ss",
"time": "one hour",
"frequency": "2"
},
{
"sub": "ms",
"time": "two hours",
"frequency": "5"
}
]
}
]
}
Now I want to delete
{
"sub": "edc",
"time": "one hour",
"frequency": "3"
}
using "sub":"edc" from the following collection
I want the query to perform changes in mongo db
You need to use $pull, although i've not done it with nested array.
See http://docs.mongodb.org/manual/reference/operator/pull/
Something like: (but you'll need to test it)
db.yourcoll.update( { "branch.section.sub": 'edu' }, { $pull: { "branch.section.sub": 'edu' } } )
This is a similar question:
How to remove an element from a doubly-nested array in a MongoDB document