Iterate over json in shell [duplicate] - json

How do I get jq to take json like this:
{
"host1": { "ip": "10.1.2.3" },
"host2": { "ip": "10.1.2.2" },
"host3": { "ip": "10.1.18.1" }
}
and generate this output:
host1, 10.1.2.3
host2, 10.1.2.2
host3, 10.1.18.1
I'm not interested in the formatting, I just can't figure out how to access the key name and value.

To get the top-level keys as a stream, you can use the built-in function keys[]. So one solution to your particular problem would be:
jq -r 'keys[] as $k | "\($k), \(.[$k] | .ip)"'
keys produces the key names in sorted order; if you want them in the original order, use keys_unsorted.
Another alternative, which produces keys in the original order, is:
jq -r 'to_entries[] | "\(.key), \(.value | .ip)"'
CSV and TSV output
The #csv and #tsv filters might also be worth considering here, e.g.
jq -r 'to_entries[] | [.key, .value.ip] | #tsv'
produces:
host1 10.1.2.3
host2 10.1.2.2
host3 10.1.18.1
Embedded objects
If the keys of interest are embedded as in the following example, the jq filter would have to be modified along the lines shown.
Input:
{
"myhosts": {
"host1": { "ip": "10.1.2.3" },
"host2": { "ip": "10.1.2.2" },
"host3": { "ip": "10.1.18.1" }
}
}
Modification:
jq -r '.myhosts | keys[] as $k | "\($k), \(.[$k] | .ip)"'

Came across very elegant solution
jq 'with_entries(.value |= .ip)'
Which ouputs
{
"host1": "10.1.2.3",
"host2": "10.1.2.2",
"host3": "10.1.18.1"
}
Here is the jqplay snippet to play with: https://jqplay.org/s/Jb_fnBveMQ
The function with_entries converts each object in the list of objects to Key/Value-pair, thus we can access .key or .value respectively, we're updating (overwriting) every KV-item .value with the field .ip by using update |= operator

Related

Extract value inside a matching block using JQ in a nested array [duplicate]

I have the following json file:
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"name": "Jack",
"location": "Whereever"
}
}
I am using jq and want to get the "name" elements of the objects where 'location' is 'Stockholm'.
I know I can get all names by
cat json | jq .[] | jq ."name"
"Jack"
"Walt"
"Donald"
But I can't figure out how to print only certain objects, given the value of a sub key (here: "location" : "Stockholm").
Adapted from this post on Processing JSON with jq, you can use the select(bool) like this:
$ jq '.[] | select(.location=="Stockholm")' json
{
"location": "Stockholm",
"name": "Walt"
}
{
"location": "Stockholm",
"name": "Donald"
}
To obtain a stream of just the names:
$ jq '.[] | select(.location=="Stockholm") | .name' json
produces:
"Donald"
"Walt"
To obtain a stream of corresponding (key name, "name" attribute) pairs, consider:
$ jq -c 'to_entries[]
| select (.value.location == "Stockholm")
| [.key, .value.name]' json
Output:
["FOO","Donald"]
["BAR","Walt"]
I had a similar related question: What if you wanted the original object format back (with key names, e.g. FOO, BAR)?
Jq provides to_entries and from_entries to convert between objects and key-value pair arrays. That along with map around the select
These functions convert between an object and an array of key-value
pairs. If to_entries is passed an object, then for each k: v entry in
the input, the output array includes {"key": k, "value": v}.
from_entries does the opposite conversion, and with_entries(foo) is a
shorthand for to_entries | map(foo) | from_entries, useful for doing
some operation to all keys and values of an object. from_entries
accepts key, Key, name, Name, value and Value as keys.
jq15 < json 'to_entries | map(select(.value.location=="Stockholm")) | from_entries'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Using the with_entries shorthand, this becomes:
jq15 < json 'with_entries(select(.value.location=="Stockholm"))'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Just try this one as a full copy paste in the shell and you will grasp it.
# pass the multiline string to the jq, use the jq to
# select the attribute named "card_id"
# ONLY if its neighbour attribute
# named "card_id_type" has the "card_id_type-01" value.
# jq -r means give me ONLY the value of the jq query no quotes aka raw
cat << EOF | \
jq -r '.[]| select (.card_id_type == "card_id_type-01")|.card_id'
[
{ "card_id": "id-00", "card_id_type": "card_id_type-00"},
{ "card_id": "id-01", "card_id_type": "card_id_type-01"},
{ "card_id": "id-02", "card_id_type": "card_id_type-02"}
]
EOF
# this ^^^ MUST start first on the line - no whitespace there !!!
# outputs:
# id-01
or with an aws cli command
# list my vpcs or
# list the values of the tags which names are "Name"
aws ec2 describe-vpcs | jq -r '.| .Vpcs[].Tags[]
|select (.Key == "Name") | .Value'|sort -nr
Note that you could move up and down in the hierarchy both during the filtering phase and during the selecting phase :
kubectl get services --all-namespaces -o json | jq -r '
.items[] | select( .metadata.name
| contains("my-srch-string")) |
{ name: .metadata.name, ns: .metadata.namespace
, nodePort: .spec.ports[].nodePort
, port: .spec.ports[].port}
'

jq : print key and value for each entry in nested object

This is JSON Object
{
"success": true,
"terms": "https://coinlayer.com/terms",
"privacy": "https://coinlayer.com/privacy",
"timestamp": 1620244806,
"target": "USD",
"rates": {
"611": 0.389165,
"ABC": 59.99,
"ACP": 0.014931,
"ACT": 0.021098,
"ACT*": 0.017178,
"ADA": 1.460965
}
}
I require this type of output:
611,0.389165
ABC,59.99
ACP,0.014931
ACT,0.021098
ACT*,0.017178
ADA,1.460965
Can somebody help me figure out doing it preferably with jq, shell script or command.
You can use #csv to generate CSV output from arrays, and to_entries to break up the object's elements into said arrays:
$ jq -r '.rates | to_entries[] | [ .key, .value ] | #csv' input.json
"611",0.389165
"ABC",59.99
"ACP",0.014931
"ACT",0.021098
"ACT*",0.017178
"ADA",1.460965

Transforming a list containing key/value strings with jq

So, I basically have a file test.json
[
"Name=TestName",
"Tag=TestTag"
]
Which I'd like to transform into
[
{
"ParameterKey": "Name",
"ParameterValue": "TestName",
},
{
"ParameterKey": "Tag",
"ParameterValue": "TestTag",
}
]
With jq. Any idea?
You don't need to use split() call twice but just once and access the results directly with the Array/Object Value Iterator: .[] and specifying the index inside
jq -n '[ inputs[] | split("=") | {ParameterKey: .[0], ParameterValue: .[1]} ]'
You can try JQ Play
I tried with the following jq. It should work as long as you are sure of the format of the array.
[.[] | {ParameterKey: split("=")[0], ParameterValue: split("=")[1]}]
If you are using from terminal, you can use the following option
cat test.json | jq '[.[] | {ParameterKey: split("=")[0], ParameterValue: split("=")[1]}]'

Printing all keys and values in a single line after sorting the keys

I've a folder with more than 1000 request logs (generated per hour/day) which are in the following format:
[
{
"input": {
"random_param_name_1": "random_value_1",
"random_param_name_2": "random_value_2",
"random_param_name_3": "random_value_3",
"random_param_name_4": "random_value_4"
},
"output": {
"some_key_we_dont_care_about": "some_value_we_dont_care_about"
},
"status_code": 200
},
{
"input": {
"random_param_name_1": "random_value_1",
"random_param_name_4": "random_value_4",
"random_param_name_3": "random_value_3",
"random_param_name_5": "random_value_5"
},
"output": {
"some_key_we_dont_care_about": "some_value_we_dont_care_about"
},
"status_code": 200
}
]
And I need to find all the input requests that are unique. For this, I need to do two things:
sort the keys in input as different inputs might have same keys but in different order
print all the key and value in a single line, so that I can pipe the output to sort | uniq to get all the unique input combinations.
Please note that the input keys are random, most existing questions in stackoverflow of the similar kind, know the keys in advance, but that's not the case here.
I can print the key and values like this:
jq -r 'keys[] as $k | "\($k):(.[$k])"'
but they end up being on new lines.
to summarise, for the above json, I need a magic_expression
$ jq 'magic_expression' log.json
that will return
"random_param_name_1":"random_value_1","random_param_name_2":"random_value_2","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4"
"random_param_name_1":"random_value_1","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4","random_param_name_5":"random_value_5"
Here is a "magic expression" to get you started.
It uses to_entries to make the objects appearing in .input more managable.
def format: "\"\(.key)\":\"\(.value)\"" ;
map(.input) | unique | map(to_entries)[] | map(format) | join(",")
When run with -r / --raw-output it produces
"random_param_name_1":"random_value_1","random_param_name_2":"random_value_2","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4"
"random_param_name_1":"random_value_1","random_param_name_4":"random_value_4","random_param_name_3":"random_value_3","random_param_name_5":"random_value_5"
Try it online!
EDIT: if as customcommander points out you want the keys to be sorted you can move the format before the unique. e.g.
def format: "\"\(.key)\":\"\(.value)\"" ;
map(.input | to_entries | map(format) | sort ) | unique[] | join(",")
which produces
"random_param_name_1":"random_value_1","random_param_name_2":"random_value_2","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4"
"random_param_name_1":"random_value_1","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4","random_param_name_5":"random_value_5"
when run with -r / --raw-output
Try it online!
Consider this:
/workspaces # jq 'map(.input)' data.json
[
{
"random_param_name_1": "random_value_1",
"random_param_name_2": "random_value_2",
"random_param_name_3": "random_value_3",
"random_param_name_4": "random_value_4"
},
{
"random_param_name_1": "random_value_1",
"random_param_name_4": "random_value_4",
"random_param_name_3": "random_value_3",
"random_param_name_5": "random_value_5"
}
]
You can sort the keys of each object with --sort-keys:
/workspaces # jq --sort-keys 'map(.input)' data.json
[
{
"random_param_name_1": "random_value_1",
"random_param_name_2": "random_value_2",
"random_param_name_3": "random_value_3",
"random_param_name_4": "random_value_4"
},
{
"random_param_name_1": "random_value_1",
"random_param_name_3": "random_value_3",
"random_param_name_4": "random_value_4",
"random_param_name_5": "random_value_5"
}
]
Then pipe this into another jq filter:
/workspaces # jq --sort-keys 'map(.input)' data.json | jq -r 'map(to_entries)[] | map("\"\(.key)\":\"\(.value)\"") | join(",")'
"random_param_name_1":"random_value_1","random_param_name_2":"random_value_2","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4"
"random_param_name_1":"random_value_1","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4","random_param_name_5":"random_value_5"
I need to find all the input requests that are unique.
This can be done within jq, without any sorting of keys, since jq's == operator ignores the key order. For example, the following will produce the unique input requests in their original form (i.e. without the keys being sorted):
map(.input)
| group_by(.)
| map(.[0])
Since group_by uses ==, uniqueness is guaranteed.
If you really want the keys to be sorted, then you could use the -S command-line option:
jq -S -f program.jq input.json
And if for some reason you really want the non-standard output format, you could use the following modification of the above program:
map(.input)
| group_by(.)
| map(.[0])
| .[]
| . as $in
| [ keys[] as $k | "\"\($k)\":\"\($in[$k])\"" ] | join(",")
With your sample input, this last produces:
"random_param_name_1":"random_value_1","random_param_name_2":"random_value_2","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4"
"random_param_name_1":"random_value_1","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4","random_param_name_5":"random_value_5"

jq: print key and value for each entry in an object

How do I get jq to take json like this:
{
"host1": { "ip": "10.1.2.3" },
"host2": { "ip": "10.1.2.2" },
"host3": { "ip": "10.1.18.1" }
}
and generate this output:
host1, 10.1.2.3
host2, 10.1.2.2
host3, 10.1.18.1
I'm not interested in the formatting, I just can't figure out how to access the key name and value.
To get the top-level keys as a stream, you can use the built-in function keys[]. So one solution to your particular problem would be:
jq -r 'keys[] as $k | "\($k), \(.[$k] | .ip)"'
keys produces the key names in sorted order; if you want them in the original order, use keys_unsorted.
Another alternative, which produces keys in the original order, is:
jq -r 'to_entries[] | "\(.key), \(.value | .ip)"'
CSV and TSV output
The #csv and #tsv filters might also be worth considering here, e.g.
jq -r 'to_entries[] | [.key, .value.ip] | #tsv'
produces:
host1 10.1.2.3
host2 10.1.2.2
host3 10.1.18.1
Embedded objects
If the keys of interest are embedded as in the following example, the jq filter would have to be modified along the lines shown.
Input:
{
"myhosts": {
"host1": { "ip": "10.1.2.3" },
"host2": { "ip": "10.1.2.2" },
"host3": { "ip": "10.1.18.1" }
}
}
Modification:
jq -r '.myhosts | keys[] as $k | "\($k), \(.[$k] | .ip)"'
Came across very elegant solution
jq 'with_entries(.value |= .ip)'
Which ouputs
{
"host1": "10.1.2.3",
"host2": "10.1.2.2",
"host3": "10.1.18.1"
}
Here is the jqplay snippet to play with: https://jqplay.org/s/Jb_fnBveMQ
The function with_entries converts each object in the list of objects to Key/Value-pair, thus we can access .key or .value respectively, we're updating (overwriting) every KV-item .value with the field .ip by using update |= operator