I'm trying to parse some json with jq that looks like this:
{
"currentServerId": 1,
"isCurrentServerAvailable": true,
"isClusterOwner": false,
"clusterOwnerServerId": "2",
"numInvalidEventMessages": "0",
"numInvalidoperationalStateMessages": "0",
"servers": {
"1": {
"isAvailableViaServerObjective": true,
"eventMessagesPendingInDb": "0",
"neo4jClusterRole": "slave",
"monitoredServerHealth": null
},
"2": {
"isAvailableViaServerObjective": true,
"eventMessagesPendingInDb": "0",
"neo4jClusterRole": "master",
"monitoredServerHealth": {
"serverId": 2,
"healthState": "ALIVE",
"healthCount": 1,
"serverIdAsString": "2",
"serverUid": "*"
}
},
"3": {
"isAvailableViaServerObjective": true,
"eventMessagesPendingInDb": "0",
"neo4jClusterRole": "slave",
"monitoredServerHealth": null
}
}
}
I can obtain the value I need with a filter like this:
# cat test.json | jq .'servers."2".monitoredServerHealth.healthState'
"ALIVE"
If I pass in a single variable as a key it works:
# echo $var
servers
# cat json.tst | jq --arg keyvar "$var" '.[$keyvar]'
{
"1": {
"isAvailableViaServerObjective": true,
"eventMessagesPendingInDb": "0",
"neo4jClusterRole": "slave",
"monitoredServerHealth": null
},
<omitted the rest for brevity>
When I try to do the same for a nested block (where one level of the key is from a variable), it fails. I've tried a number of variations but they all fail.
What is the correct syntax?
# echo $var
"2"
# cat test.json | jq --arg keyvar "$var" '.servers.$[keyvar].monitoredServerHealth.healthState'
jq: error: syntax error, unexpected '$', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
.servers.$[keyvar].monitoredServerHealth.healthState
jq: 1 compile error
Split up the path into two parts:
$ jq --arg keyvar "$var" '.servers | .[$keyvar].monitoredServerHealth.healthState' input.json
"ALIVE"
From the documentation:
A filter of the form .foo.bar is equivalent to .foo|.bar.
I don't know why it doesn't work combined as .servers.[$keyvar]; might be a parser bug or something. At least the workaround is trivial.
Another way uses string interpolation:
jq --arg keyvar "$var" '.servers."\($keyvar)".monitoredServerHealth.healthState' input.json
Related
I have the following json file:
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"name": "Jack",
"location": "Whereever"
}
}
I am using jq and want to get the "name" elements of the objects where 'location' is 'Stockholm'.
I know I can get all names by
cat json | jq .[] | jq ."name"
"Jack"
"Walt"
"Donald"
But I can't figure out how to print only certain objects, given the value of a sub key (here: "location" : "Stockholm").
Adapted from this post on Processing JSON with jq, you can use the select(bool) like this:
$ jq '.[] | select(.location=="Stockholm")' json
{
"location": "Stockholm",
"name": "Walt"
}
{
"location": "Stockholm",
"name": "Donald"
}
To obtain a stream of just the names:
$ jq '.[] | select(.location=="Stockholm") | .name' json
produces:
"Donald"
"Walt"
To obtain a stream of corresponding (key name, "name" attribute) pairs, consider:
$ jq -c 'to_entries[]
| select (.value.location == "Stockholm")
| [.key, .value.name]' json
Output:
["FOO","Donald"]
["BAR","Walt"]
I had a similar related question: What if you wanted the original object format back (with key names, e.g. FOO, BAR)?
Jq provides to_entries and from_entries to convert between objects and key-value pair arrays. That along with map around the select
These functions convert between an object and an array of key-value
pairs. If to_entries is passed an object, then for each k: v entry in
the input, the output array includes {"key": k, "value": v}.
from_entries does the opposite conversion, and with_entries(foo) is a
shorthand for to_entries | map(foo) | from_entries, useful for doing
some operation to all keys and values of an object. from_entries
accepts key, Key, name, Name, value and Value as keys.
jq15 < json 'to_entries | map(select(.value.location=="Stockholm")) | from_entries'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Using the with_entries shorthand, this becomes:
jq15 < json 'with_entries(select(.value.location=="Stockholm"))'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Just try this one as a full copy paste in the shell and you will grasp it.
# pass the multiline string to the jq, use the jq to
# select the attribute named "card_id"
# ONLY if its neighbour attribute
# named "card_id_type" has the "card_id_type-01" value.
# jq -r means give me ONLY the value of the jq query no quotes aka raw
cat << EOF | \
jq -r '.[]| select (.card_id_type == "card_id_type-01")|.card_id'
[
{ "card_id": "id-00", "card_id_type": "card_id_type-00"},
{ "card_id": "id-01", "card_id_type": "card_id_type-01"},
{ "card_id": "id-02", "card_id_type": "card_id_type-02"}
]
EOF
# this ^^^ MUST start first on the line - no whitespace there !!!
# outputs:
# id-01
or with an aws cli command
# list my vpcs or
# list the values of the tags which names are "Name"
aws ec2 describe-vpcs | jq -r '.| .Vpcs[].Tags[]
|select (.Key == "Name") | .Value'|sort -nr
Note that you could move up and down in the hierarchy both during the filtering phase and during the selecting phase :
kubectl get services --all-namespaces -o json | jq -r '
.items[] | select( .metadata.name
| contains("my-srch-string")) |
{ name: .metadata.name, ns: .metadata.namespace
, nodePort: .spec.ports[].nodePort
, port: .spec.ports[].port}
'
I have the following json file:
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"name": "Jack",
"location": "Whereever"
}
}
I am using jq and want to get the "name" elements of the objects where 'location' is 'Stockholm'.
I know I can get all names by
cat json | jq .[] | jq ."name"
"Jack"
"Walt"
"Donald"
But I can't figure out how to print only certain objects, given the value of a sub key (here: "location" : "Stockholm").
Adapted from this post on Processing JSON with jq, you can use the select(bool) like this:
$ jq '.[] | select(.location=="Stockholm")' json
{
"location": "Stockholm",
"name": "Walt"
}
{
"location": "Stockholm",
"name": "Donald"
}
To obtain a stream of just the names:
$ jq '.[] | select(.location=="Stockholm") | .name' json
produces:
"Donald"
"Walt"
To obtain a stream of corresponding (key name, "name" attribute) pairs, consider:
$ jq -c 'to_entries[]
| select (.value.location == "Stockholm")
| [.key, .value.name]' json
Output:
["FOO","Donald"]
["BAR","Walt"]
I had a similar related question: What if you wanted the original object format back (with key names, e.g. FOO, BAR)?
Jq provides to_entries and from_entries to convert between objects and key-value pair arrays. That along with map around the select
These functions convert between an object and an array of key-value
pairs. If to_entries is passed an object, then for each k: v entry in
the input, the output array includes {"key": k, "value": v}.
from_entries does the opposite conversion, and with_entries(foo) is a
shorthand for to_entries | map(foo) | from_entries, useful for doing
some operation to all keys and values of an object. from_entries
accepts key, Key, name, Name, value and Value as keys.
jq15 < json 'to_entries | map(select(.value.location=="Stockholm")) | from_entries'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Using the with_entries shorthand, this becomes:
jq15 < json 'with_entries(select(.value.location=="Stockholm"))'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Just try this one as a full copy paste in the shell and you will grasp it.
# pass the multiline string to the jq, use the jq to
# select the attribute named "card_id"
# ONLY if its neighbour attribute
# named "card_id_type" has the "card_id_type-01" value.
# jq -r means give me ONLY the value of the jq query no quotes aka raw
cat << EOF | \
jq -r '.[]| select (.card_id_type == "card_id_type-01")|.card_id'
[
{ "card_id": "id-00", "card_id_type": "card_id_type-00"},
{ "card_id": "id-01", "card_id_type": "card_id_type-01"},
{ "card_id": "id-02", "card_id_type": "card_id_type-02"}
]
EOF
# this ^^^ MUST start first on the line - no whitespace there !!!
# outputs:
# id-01
or with an aws cli command
# list my vpcs or
# list the values of the tags which names are "Name"
aws ec2 describe-vpcs | jq -r '.| .Vpcs[].Tags[]
|select (.Key == "Name") | .Value'|sort -nr
Note that you could move up and down in the hierarchy both during the filtering phase and during the selecting phase :
kubectl get services --all-namespaces -o json | jq -r '
.items[] | select( .metadata.name
| contains("my-srch-string")) |
{ name: .metadata.name, ns: .metadata.namespace
, nodePort: .spec.ports[].nodePort
, port: .spec.ports[].port}
'
I need a jq command to replace array values with non-json standard values i.e. in the following json input I need to replace the "webOrigins" array values with a non-json value, which is a Jinja2 template variable replacement as per the second json block below.
Input (example.json)
{
"clients": [
{
"clientId": "abc",
"webOrigins": [ "/", "/api" ]
},
{
"clientId": "xyz",
"webOrigins": [ ]
}
]
}
Desired Output
{
"clients": [
{
"clientId": "abc",
"webOrigins": {{clients.abc.webOrigins}}
},
{
"clientId": "xyz",
"webOrigins": {{clients.xyz.webOrigins}}
}
]
}
My current attempt of shell script calling jq to loop through the input json and replace with template variable is this
for clientId in $(jq -r '.clients[] | .clientId' example.json)
do
jq '(.clients[] | select(.clientId == "'${clientId}'") | .webOrigins) |= {{ clients['\'${clientId}\''].webOrigins | default([]) }}' example.json > tmp.j2; mv -f tmp.j2 example.json
done
But fails with the error:
jq: error: syntax error, unexpected '{' (Unix shell quoting issues?) at <top-level>, line 1:
(.clients[] | select(.clientId == "abc") | .webOrigins) |= {{ clients['abc'].webOrigins | default([]) }}
jq: 1 compile error
Of course if I add double quotes around the template variable to make the replacement variable valid json the script works but I need the value to not have quotes.
It would be impractical to implement a jq-only solution, but jq can certainly help, e.g. the following works in your case:
jq '.clients[] |= (.webOrigins = "{{clients.\(.clientId).webOrigins}}")' |
sed '/"webOrigins":/ { s/"[{][{]/{{/; s/[}][}]"$/}}/; }'
I have a while loop with two variables I have to merge into a single piece of JSON, like so:
#!/bin/bash
while read -r from to
do
# BONUS: Solution would ideally require no quoting at this point
echo { \"From\": \"$from\", \"To\": \"$to\" }
done << EOF
foo bar
what ever
EOF
Which currently outputs invalid JSON:
{ "From": "foo", "To": "bar" }
{ "From": "what", "To": "ever" }
What's the simplest I can create valid JSON like:
[
{ "From": "foo", "To": "bar" },
{ "From": "what", "To": "ever" }
]
I looked at jq but I couldn't figure out how to do it. I'm not looking to do it in shell ideally because I feel adding commas and such is a bit ugly.
With jq:
$ jq -nR '[inputs | split(" ") | {"From": .[0], "To": .[1]}]' <<EOF
foo bar
what ever
EOF
[
{
"From": "foo",
"To": "bar"
},
{
"From": "what",
"To": "ever"
}
]
-n tells jq to not read any input; -R is for raw input so it doesn't expect JSON.
The input is read with inputs, resulting in one string per input line:
$ jq -nR 'inputs' <<EOF
foo bar
what ever
EOF
"foo bar"
"what ever"
These are then split into arrays of words:
$ jq -nR 'inputs | split(" ")' <<EOF
foo bar
what ever
EOF
[
"foo",
"bar"
]
[
"what",
"ever"
]
From this, we construct the objects:
$ jq -nR 'inputs | split(" ") | {"From": .[0], "To": .[1]}' <<EOF
foo bar
what ever
EOF
{
"From": "foo",
"To": "bar"
}
{
"From": "what",
"To": "ever"
}
And finally, we wrap everything in [] to get the final output shown first.
The more intuitive approach of splitting input directly fails because wrapping everything in [] results in one array per input line:
$ jq -R '[split(" ") | { "From": .[0], "To": .[1] }]' <<EOF
foo bar
what ever
EOF
[
{
"From": "foo",
"To": "bar"
}
]
[
{
"From": "what",
"To": "ever"
}
]
Hence the somewhat cumbersome -n/inputs. Notice that inputs requires jq version 1.5.
Here's an all-jq solution that assumes the "From" and "To" values are presented exactly as in your example:
jq -R -n '[inputs | split(" ") | {From: .[0], To: .[1]}]'
You might want to handle additional spaces using gsub/2.
If your jq does not have inputs then you can use this incantation:
jq -R -s 'split("\n")
| map(select(length>1) | split(" ") | {From: .[0], To: .[1]})'
Or you could just pipe the output from your while-loop into jq -s.
I have JSON like this that I'm parsing with jq:
{
"data": [
{
"item": {
"name": "string 1"
},
"item": {
"name": "string 2"
},
"item": {
"name": "string 3"
}
}
]
}
...and I'm trying to get "string 1" "string 2" and "string 3" into a Bash array, but I can't find a solution that ignores the whitespace in them. Is there a method in jq that I'm missing, or perhaps an elegant solution in Bash for it?
Current method:
json_names=$(cat file.json | jq ".data[] .item .name")
read -a name_array <<< $json_names
The below assume your JSON text is in a string named s. That is:
s='{
"data": [
{
"item1": {
"name": "string 1"
},
"item2": {
"name": "string 2"
},
"item3": {
"name": "string 3"
}
}
]
}'
Unfortunately, both of the below will misbehave with strings containing literal newlines; since jq doesn't have support for NUL-delimited output, this is difficult to work around.
On bash 4 (with slightly sloppy error handling, but tersely):
readarray -t name_array < <(jq -r '.data[] | .[] | .name' <<<"$s")
...or on bash 3.x or newer (with very comprehensive error handling, but verbosely):
# -d '' tells read to process up to a NUL, and will exit with a nonzero exit status if that
# NUL is not seen; thus, this causes the read to pass through any error which occurred in
# jq.
IFS=$'\n' read -r -d '' -a name_array \
< <(jq -r '.data[] | .[] | .name' <<<"$s" && printf '\0')
This populates a bash array, contents of which can be displayed with:
declare -p name_array
Arrays are assigned in the form:
NAME=(VALUE1 VALUE2 ... )
where NAME is the name of the variable, VALUE1, VALUE2, and the rest are fields separated with characters that are present in the $IFS (input field separator) variable.
Since jq outputs the string values as lines (sequences separated with the new line character), then you can temporarily override $IFS, e.g.:
# Disable globbing, remember current -f flag value
[[ "$-" == *f* ]] || globbing_disabled=1
set -f
IFS=$'\n' a=( $(jq --raw-output '.data[].item.name' file.json) )
# Restore globbing
test -n "$globbing_disabled" && set +f
The above will create an array of three items for the following file.json:
{
"data": [
{"item": {
"name": "string 1"
}},
{"item": {
"name": "string 2"
}},
{"item": {
"name": "string 3"
}}
]
}
The following shows how to create a bash array consisting of arbitrary JSON texts produced by a run of jq.
In the following, I'll assume input.json is a file with the following:
["string 1", "new\nline", {"x": 1}, ["one\ttab", 4]]
With this input, the jq filter .[] produces four JSON texts -- two JSON strings, a JSON object, and a JSON array.
The following bash script can then be used to set x to be a bash array of the JSON texts:
#!/bin/bash
x=()
while read -r value
do
x+=("$value")
done < <(jq -c '.[]' input.json)
For example, adding this bash expression to the script:
for a in "${x[#]}" ; do echo a="$a"; done
would yield:
a="string 1"
a="new\nline"
a={"x":1}
a=["one\ttab",4]
Notice how (encoded) newlines and (encoded) tabs are handled properly.