Get the index of the array element in JSON with jq - json

I have the following type of json:
{
"foo": "hello",
"bar": [
{
"key": "k1",
"val": "v1"
},
{
"key": "k2",
"val": "v2"
},
{
"key": "k3",
"val": "v3"
}
]
}
I want to output the following:
"hello", 1, "k1", "v1"
"hello", 2, "k2", "v2"
"hello", 3, "k3", "v3"
I am using jq to tranform this and the answer should also be with a jq transformation.
I am currently at:
echo '{"foo": "hello","bar": [{"key": "k1","val": "v1"},{"key": "k2","val": "v2"},{"key": "k3","val": "v3"} ]}' | jq -c -r '.bar[] as $b | [.foo, ($b | .key, .val)] | #csv'
Which gives me:
"hello","k1","v1"
"hello","k2","v2"
"hello","k3","v3"
How can I also get the index to show of the array element being parsed?

You could convert the array to entries to access the index and the value. Then you can build out the CSV rows.
$ jq -r '[.foo] + (.bar | to_entries[] | [.key+1,.value.key,.value.val]) | #csv' input.json
"hello",1,"k1","v1"
"hello",2,"k2","v2"
"hello",3,"k3","v3"

Assuming you have access to jq 1.5 and that the key/val keys are presented in that order:
jq -r '.foo as $foo
| foreach .bar[] as $i (0; .+1; [$foo, .] + [$i[]])
| #csv'
would produce:
"hello",1,"k1","v1"
"hello",2,"k2","v2"
"hello",3,"k3","v3"
The -r option is often used with #csv to convert the JSON string that would otherwise be produced by #csv into a comma-separated list of values.
If you really want to join with ", ", then it's a bit messier, but if you're not worried about the functionality that #csv provides, here's one way:
$ jq -r '"\"\(.foo)\"" as $foo
| foreach .bar[] as $i
(0; .+1; "\($foo), \(.), \($i | map("\"\(.)\"")|join(", "))")'
This produces:
"hello", 1, "k1", "v1"
"hello", 2, "k2", "v2"
"hello", 3, "k3", "v3"
If your jq does not have foreach then you could similarly use reduce, but it might be easier to upgrade.

Related

Extract value inside a matching block using JQ in a nested array [duplicate]

I have the following json file:
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"name": "Jack",
"location": "Whereever"
}
}
I am using jq and want to get the "name" elements of the objects where 'location' is 'Stockholm'.
I know I can get all names by
cat json | jq .[] | jq ."name"
"Jack"
"Walt"
"Donald"
But I can't figure out how to print only certain objects, given the value of a sub key (here: "location" : "Stockholm").
Adapted from this post on Processing JSON with jq, you can use the select(bool) like this:
$ jq '.[] | select(.location=="Stockholm")' json
{
"location": "Stockholm",
"name": "Walt"
}
{
"location": "Stockholm",
"name": "Donald"
}
To obtain a stream of just the names:
$ jq '.[] | select(.location=="Stockholm") | .name' json
produces:
"Donald"
"Walt"
To obtain a stream of corresponding (key name, "name" attribute) pairs, consider:
$ jq -c 'to_entries[]
| select (.value.location == "Stockholm")
| [.key, .value.name]' json
Output:
["FOO","Donald"]
["BAR","Walt"]
I had a similar related question: What if you wanted the original object format back (with key names, e.g. FOO, BAR)?
Jq provides to_entries and from_entries to convert between objects and key-value pair arrays. That along with map around the select
These functions convert between an object and an array of key-value
pairs. If to_entries is passed an object, then for each k: v entry in
the input, the output array includes {"key": k, "value": v}.
from_entries does the opposite conversion, and with_entries(foo) is a
shorthand for to_entries | map(foo) | from_entries, useful for doing
some operation to all keys and values of an object. from_entries
accepts key, Key, name, Name, value and Value as keys.
jq15 < json 'to_entries | map(select(.value.location=="Stockholm")) | from_entries'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Using the with_entries shorthand, this becomes:
jq15 < json 'with_entries(select(.value.location=="Stockholm"))'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Just try this one as a full copy paste in the shell and you will grasp it.
# pass the multiline string to the jq, use the jq to
# select the attribute named "card_id"
# ONLY if its neighbour attribute
# named "card_id_type" has the "card_id_type-01" value.
# jq -r means give me ONLY the value of the jq query no quotes aka raw
cat << EOF | \
jq -r '.[]| select (.card_id_type == "card_id_type-01")|.card_id'
[
{ "card_id": "id-00", "card_id_type": "card_id_type-00"},
{ "card_id": "id-01", "card_id_type": "card_id_type-01"},
{ "card_id": "id-02", "card_id_type": "card_id_type-02"}
]
EOF
# this ^^^ MUST start first on the line - no whitespace there !!!
# outputs:
# id-01
or with an aws cli command
# list my vpcs or
# list the values of the tags which names are "Name"
aws ec2 describe-vpcs | jq -r '.| .Vpcs[].Tags[]
|select (.Key == "Name") | .Value'|sort -nr
Note that you could move up and down in the hierarchy both during the filtering phase and during the selecting phase :
kubectl get services --all-namespaces -o json | jq -r '
.items[] | select( .metadata.name
| contains("my-srch-string")) |
{ name: .metadata.name, ns: .metadata.namespace
, nodePort: .spec.ports[].nodePort
, port: .spec.ports[].port}
'

jq : print key and value for each entry in nested object

This is JSON Object
{
"success": true,
"terms": "https://coinlayer.com/terms",
"privacy": "https://coinlayer.com/privacy",
"timestamp": 1620244806,
"target": "USD",
"rates": {
"611": 0.389165,
"ABC": 59.99,
"ACP": 0.014931,
"ACT": 0.021098,
"ACT*": 0.017178,
"ADA": 1.460965
}
}
I require this type of output:
611,0.389165
ABC,59.99
ACP,0.014931
ACT,0.021098
ACT*,0.017178
ADA,1.460965
Can somebody help me figure out doing it preferably with jq, shell script or command.
You can use #csv to generate CSV output from arrays, and to_entries to break up the object's elements into said arrays:
$ jq -r '.rates | to_entries[] | [ .key, .value ] | #csv' input.json
"611",0.389165
"ABC",59.99
"ACP",0.014931
"ACT",0.021098
"ACT*",0.017178
"ADA",1.460965

bash: parse data from Json file [duplicate]

I have the following json file:
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"name": "Jack",
"location": "Whereever"
}
}
I am using jq and want to get the "name" elements of the objects where 'location' is 'Stockholm'.
I know I can get all names by
cat json | jq .[] | jq ."name"
"Jack"
"Walt"
"Donald"
But I can't figure out how to print only certain objects, given the value of a sub key (here: "location" : "Stockholm").
Adapted from this post on Processing JSON with jq, you can use the select(bool) like this:
$ jq '.[] | select(.location=="Stockholm")' json
{
"location": "Stockholm",
"name": "Walt"
}
{
"location": "Stockholm",
"name": "Donald"
}
To obtain a stream of just the names:
$ jq '.[] | select(.location=="Stockholm") | .name' json
produces:
"Donald"
"Walt"
To obtain a stream of corresponding (key name, "name" attribute) pairs, consider:
$ jq -c 'to_entries[]
| select (.value.location == "Stockholm")
| [.key, .value.name]' json
Output:
["FOO","Donald"]
["BAR","Walt"]
I had a similar related question: What if you wanted the original object format back (with key names, e.g. FOO, BAR)?
Jq provides to_entries and from_entries to convert between objects and key-value pair arrays. That along with map around the select
These functions convert between an object and an array of key-value
pairs. If to_entries is passed an object, then for each k: v entry in
the input, the output array includes {"key": k, "value": v}.
from_entries does the opposite conversion, and with_entries(foo) is a
shorthand for to_entries | map(foo) | from_entries, useful for doing
some operation to all keys and values of an object. from_entries
accepts key, Key, name, Name, value and Value as keys.
jq15 < json 'to_entries | map(select(.value.location=="Stockholm")) | from_entries'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Using the with_entries shorthand, this becomes:
jq15 < json 'with_entries(select(.value.location=="Stockholm"))'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Just try this one as a full copy paste in the shell and you will grasp it.
# pass the multiline string to the jq, use the jq to
# select the attribute named "card_id"
# ONLY if its neighbour attribute
# named "card_id_type" has the "card_id_type-01" value.
# jq -r means give me ONLY the value of the jq query no quotes aka raw
cat << EOF | \
jq -r '.[]| select (.card_id_type == "card_id_type-01")|.card_id'
[
{ "card_id": "id-00", "card_id_type": "card_id_type-00"},
{ "card_id": "id-01", "card_id_type": "card_id_type-01"},
{ "card_id": "id-02", "card_id_type": "card_id_type-02"}
]
EOF
# this ^^^ MUST start first on the line - no whitespace there !!!
# outputs:
# id-01
or with an aws cli command
# list my vpcs or
# list the values of the tags which names are "Name"
aws ec2 describe-vpcs | jq -r '.| .Vpcs[].Tags[]
|select (.Key == "Name") | .Value'|sort -nr
Note that you could move up and down in the hierarchy both during the filtering phase and during the selecting phase :
kubectl get services --all-namespaces -o json | jq -r '
.items[] | select( .metadata.name
| contains("my-srch-string")) |
{ name: .metadata.name, ns: .metadata.namespace
, nodePort: .spec.ports[].nodePort
, port: .spec.ports[].port}
'

compare 2 json arrays and return the difference

We have a custom CD Pipeline Tool, which unfortunately does not version the deployment parameters. So I put these in a Bitbucket Repo as a json file and validate them against a REST API of this CD Tool.
So I have 2 json arrays, which are structurally the same, but may contain different objects or values in these objects. I want to compare them to see if they are different and what is different.
So far, I used the solution from here:
Using jq or alternative command line tools to diff JSON files
So I have put this in my code:
jq --argjson a "${bb_cfg}" --argjson b "${cd_tool_cfg}" -n 'def post_recurse(f): def r: (f | select(. != null) | r), .; r; def post_recurse: post_recurse(.[]?); ($a | (post_recurse | arrays) |= sort) as $a | ($b | (post_recurse | arrays) |= sort) as $b | $a == $b'
now I get a true if they are identical or false if 2 jsons have differences, but I do not know what is different.
I tried to do this with this if I get false back:
diff --suppress-common-lines -y <(jq . -S <<< "${bb_cfg}") <(jq . -S <<< "${cd_tool_cfg}")
Input $bb_cfg:
[{
"key": "IGNORE_VALIDATION_ERROR",
"value": "true",
"tags": []
},
{
"key": "BB_CFG_REPO_NAME",
"value": "cd-tool-cfg",
"tags": []
}]
Input $cd_tool_cfg
[{
"key": "IGNORE_VALIDATION_ERROR",
"value": "false",
"tags": []
},
{
"key": "BB_CFG_REPO_NAME",
"value": "cd-tool-cfg",
"tags": []
}]
which works partly, because if only the value is different, the output is like this:
"value": "true" | "value": "false"
so I do not get the whole json object here to quickly find out what parameter is different.
What I eventually want is to get something like this:
{
"key": "IGNORE_VALIDATION_ERROR",
"value": "true",
"tags": []
}
{
"key": "IGNORE_VALIDATION_ERROR",
"value": "false",
"tags": []
}
where I can store this in a variable in my bash script and transform this in an output I can use.
You could use jq's -c or --compact-output option:
diff <(jq -c .[] <<<"$bb_cfg") <(jq -c .[] <<<"$cd_tool_cfg")
1c1
< {"key":"IGNORE_VALIDATION_ERROR","value":"true","tags":[]}
---
> {"key":"IGNORE_VALIDATION_ERROR","value":"false","tags":[]}
The -c option will simply output a json with each array member on a separate line.
The following command will give you something like you requested:
diff --old-line-format="%L" --unchanged-line-format="" --new-line-format="%L" <(jq -c .[] <<<"$bb_cfg") <(jq -c .[] <<<"$cd_tool_cfg") | jq
will output:
{
"key": "IGNORE_VALIDATION_ERROR",
"value": "true",
"tags": []
}
{
"key": "IGNORE_VALIDATION_ERROR",
"value": "false",
"tags": []
}

Select objects based on value of variable in object using jq

I have the following json file:
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"name": "Jack",
"location": "Whereever"
}
}
I am using jq and want to get the "name" elements of the objects where 'location' is 'Stockholm'.
I know I can get all names by
cat json | jq .[] | jq ."name"
"Jack"
"Walt"
"Donald"
But I can't figure out how to print only certain objects, given the value of a sub key (here: "location" : "Stockholm").
Adapted from this post on Processing JSON with jq, you can use the select(bool) like this:
$ jq '.[] | select(.location=="Stockholm")' json
{
"location": "Stockholm",
"name": "Walt"
}
{
"location": "Stockholm",
"name": "Donald"
}
To obtain a stream of just the names:
$ jq '.[] | select(.location=="Stockholm") | .name' json
produces:
"Donald"
"Walt"
To obtain a stream of corresponding (key name, "name" attribute) pairs, consider:
$ jq -c 'to_entries[]
| select (.value.location == "Stockholm")
| [.key, .value.name]' json
Output:
["FOO","Donald"]
["BAR","Walt"]
I had a similar related question: What if you wanted the original object format back (with key names, e.g. FOO, BAR)?
Jq provides to_entries and from_entries to convert between objects and key-value pair arrays. That along with map around the select
These functions convert between an object and an array of key-value
pairs. If to_entries is passed an object, then for each k: v entry in
the input, the output array includes {"key": k, "value": v}.
from_entries does the opposite conversion, and with_entries(foo) is a
shorthand for to_entries | map(foo) | from_entries, useful for doing
some operation to all keys and values of an object. from_entries
accepts key, Key, name, Name, value and Value as keys.
jq15 < json 'to_entries | map(select(.value.location=="Stockholm")) | from_entries'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Using the with_entries shorthand, this becomes:
jq15 < json 'with_entries(select(.value.location=="Stockholm"))'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Just try this one as a full copy paste in the shell and you will grasp it.
# pass the multiline string to the jq, use the jq to
# select the attribute named "card_id"
# ONLY if its neighbour attribute
# named "card_id_type" has the "card_id_type-01" value.
# jq -r means give me ONLY the value of the jq query no quotes aka raw
cat << EOF | \
jq -r '.[]| select (.card_id_type == "card_id_type-01")|.card_id'
[
{ "card_id": "id-00", "card_id_type": "card_id_type-00"},
{ "card_id": "id-01", "card_id_type": "card_id_type-01"},
{ "card_id": "id-02", "card_id_type": "card_id_type-02"}
]
EOF
# this ^^^ MUST start first on the line - no whitespace there !!!
# outputs:
# id-01
or with an aws cli command
# list my vpcs or
# list the values of the tags which names are "Name"
aws ec2 describe-vpcs | jq -r '.| .Vpcs[].Tags[]
|select (.Key == "Name") | .Value'|sort -nr
Note that you could move up and down in the hierarchy both during the filtering phase and during the selecting phase :
kubectl get services --all-namespaces -o json | jq -r '
.items[] | select( .metadata.name
| contains("my-srch-string")) |
{ name: .metadata.name, ns: .metadata.namespace
, nodePort: .spec.ports[].nodePort
, port: .spec.ports[].port}
'