generate JSON out of command line arguments - json

I want to create JSON output with jq that looks like this:
{
"records": [
{
"id": "1234",
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
I assumed I have to twiddle with the "filter" of jq whose concept I don't fully get after reading the doc.
This is what I got so far:
$ jq --arg id 1234 \
--arg song Yesterday \
--arg artist "The Beatles" \
'.' \
<<<'{ "records" : [{ "id":"$id", "song":"$song", "artist":"$artist" }] }'
which prints
{
"records": [
{
"id" : "$id",
"song" : "$song",
"artist" : "$artist"
}
]
}
Do I modify the filter? Do I change the input?

An alternate way to your original attempt, on jq-1.6 you can use the $ARGS.positional attribute to construct your JSON from scratch
jq -n '
$ARGS.positional | {
records: [
{
id: .[0],
song: .[1],
artist: .[2]
}
]
}' --args 1234 Yesterday "The Beatles"
As for why your original attempt didn't work, looks you are not modifying your json at all, with your filter '.' you are basically just reading in and printing out "untouched". The arguments set using --arg need to be set to the object inside the filter.

You are looking for something like this:
jq --null-input \
--arg id 1234 \
--arg song Yesterday \
--arg artist "The Beatles" \
'.records[0] = {$id, $song, $artist}'
Each variable reference between curly brackets is converted to a key-value pair where its name is the key, and its value is the value. And assigning the resulting object to .records[0] forces the creation of the surrounding structure.

jq --null-input\
--argjson id 1234\
--arg song Yesterday\
--arg artist "The Beatles"\
'{ "records" : [{ $id, $song, $artist }] }'
gives
{
"records": [
{
"id": 1234,
"song": "Yesterday",
"artist": "The Beatles"
}
]
}

I think you got the JSON/JQ the wrong way round:
This should be your JQ script:
rec.jq
{
records: [
{
id: $id,
song: $song,
artist: $artist
}
]
}
And this should be your JSON (empty):
rec.json
{}
Then:
jq --arg id 123 --arg song "Yesterday" --arg artist "The Beatles" -f rec.jq rec.json
Which produces:
{
"records": [
{
"id": "123",
"song": "Yesterday",
"artist": "The Beatles"
}
]
}

Start with an empty JSON and add the missing bits:
$ jq --arg id 1234 \
--arg song Yesterday \
--arg artist "The Beatles" \
'. | .records[0].id=$id | .records[0].song=$song | .records[0].artist=$artist' \
<<<'{}'
Outputs
{
"records": [
{
"id": "1234",
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
Another, cleaner, approach based on the answer of #Inian could be
jq -n \
--arg id 1234
--arg song Yesterday
--arg artist "The Beatles"
'{records: [{id:$id, song:$song, artist:$artist}]}'

Related

Passing JSON variable to jq for manipulation [duplicate]

Data sample - sample.json (full sample: https://pastebin.com/KFkVmc2M)
{
"ip": 3301234701,
"_shodan": {
"options": {
"referrer": "7ae15507-f5cc-4353-b72e-5cc0b1c34c5e"
},
},
"hash": -1056085507,
"os": null,
"title": "WHM Login",
"opts": {
"vulns": ["!CVE-2014-0160"],
"heartbleed": "2017/08/29 09:57:30 196.196.216.13:2087 - SAFE\
}
},
"isp": "Fiber Grid Inc",
"http": {
"redirects": [],
"title": "WHM Login",
"robots": null,
"favicon": null,
"host": "196.196.216.13",
"html":
}
Script using jq which I hoped would work and haven't found a another solution yet.
cat sample.json | jq \
--arg key0 'Host' \
--arg value0 '.host' \
--arg key1 'Vulnerability' \
--arg value1 '.opts.vulns[0]' \
--arg key2 'ISP' \
--arg value2 '.isp' \
'. | .[$key0]= $value0 | .[$key1]=$value1 | .[$key2]=$value2' \
<<<'{}'
The end result I hoped, but not getting:
{
"Host": "196.196.216.13",
"Vulnerability": "!CVE-2014-0160",
"ISP": "Fiber Grid Inc"
}
Right now it just returns the object as a string and I've tried a lot of different ways to approach the problem. I am quite new to working with JSON and jq but based on what I've read so far, the solution might not be as simple as I'm wishing?
Simply put, why isn't the object being returned as a value of the sample.json object and what do I have to do, to make it work?
Thanks!
to: chepner
{
"196.196.216.13":[
"AS63119",
"Fiber Grid Inc",
"2017-08-29T06:57:22.546423",
"!CVE-2014-0160"
],
"196.196.216.14":[
"AS63119",
"Fiber Grid Inc",
"2017-08-29T06:57:22.546423",
"!CVE-2014-0160"
]
}
Here's a variant solution using the approach suggested by #CharlesDuffy:
$ config='{"Host": ["http", "host"],
"Vulnerability": ["opts", "vulns", 0],
"ISP": ["isp"]}'
$ jq --argjson config "$config" '
. as $in
| $config
| map_values( . as $p | $in | getpath($p) )' pastebin.json
Output:
{
"Host": "196.196.216.13",
"Vulnerability": "!CVE-2014-0160",
"ISP": "Fiber Grid Inc"
}
This incidentally highlights #chepner's point: the care and maintenance required for $config is no less than the care and maintenance required for the corresponding jq query, whereas the jq query language -- by design -- offers far more flexibility.
jq does not support the evaluation of jq expressions in the
way that would be required for your attempt to work. You could do some
kind of shell interpolation, but it would be better to use JSON paths,
e.g. rather than --arg value0 .host, you could write --arg value0 "host", etc. In the following, I've used getpath/1.
There is no need to prepend the jq filter with '. |'
Anyway, assuming the JSON contents of the pastebin are in pastebin.json, you could write:
jq \
--arg key0 Host \
--argjson value0 '["http","host"]' \
--arg key1 Vulnerability \
--argjson value1 '["opts", "vulns", 0]' \
--arg key2 ISP \
--argjson value2 '["isp"]' \
'. as $in
| {}
| .[$key0] = ($in|getpath($value0))
| .[$key1] = ($in|getpath($value1))
| .[$key2] = ($in|getpath($value2))' \
pastebin.json
This would have the result:
{
"Host": "196.196.216.13",
"Vulnerability": "!CVE-2014-0160",
"ISP": "Fiber Grid Inc"
}
In response to the supplementary question ("to: chepner"):
The pastebin data does not contain "196.196.216.14" so the supplementary question is unclear, but you will probably want to start with:
jq '{(.ip_str): [.asn, .isp, .timestamp, opts.vulns[0] ]}' pastebin.json
This produces:
{
"196.196.216.13": [
"AS63119",
"Fiber Grid Inc",
"2017-08-29T06:57:22.546423",
"!CVE-2014-0160"
]
}
The key point here is the pair of parentheses around .ip_str.
As mentioned elsewhere, the jq query is again so simple and straightforward that it seems doubtful that a parameterized version will offer any advantage.

Extract value inside a matching block using JQ in a nested array [duplicate]

I have the following json file:
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"name": "Jack",
"location": "Whereever"
}
}
I am using jq and want to get the "name" elements of the objects where 'location' is 'Stockholm'.
I know I can get all names by
cat json | jq .[] | jq ."name"
"Jack"
"Walt"
"Donald"
But I can't figure out how to print only certain objects, given the value of a sub key (here: "location" : "Stockholm").
Adapted from this post on Processing JSON with jq, you can use the select(bool) like this:
$ jq '.[] | select(.location=="Stockholm")' json
{
"location": "Stockholm",
"name": "Walt"
}
{
"location": "Stockholm",
"name": "Donald"
}
To obtain a stream of just the names:
$ jq '.[] | select(.location=="Stockholm") | .name' json
produces:
"Donald"
"Walt"
To obtain a stream of corresponding (key name, "name" attribute) pairs, consider:
$ jq -c 'to_entries[]
| select (.value.location == "Stockholm")
| [.key, .value.name]' json
Output:
["FOO","Donald"]
["BAR","Walt"]
I had a similar related question: What if you wanted the original object format back (with key names, e.g. FOO, BAR)?
Jq provides to_entries and from_entries to convert between objects and key-value pair arrays. That along with map around the select
These functions convert between an object and an array of key-value
pairs. If to_entries is passed an object, then for each k: v entry in
the input, the output array includes {"key": k, "value": v}.
from_entries does the opposite conversion, and with_entries(foo) is a
shorthand for to_entries | map(foo) | from_entries, useful for doing
some operation to all keys and values of an object. from_entries
accepts key, Key, name, Name, value and Value as keys.
jq15 < json 'to_entries | map(select(.value.location=="Stockholm")) | from_entries'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Using the with_entries shorthand, this becomes:
jq15 < json 'with_entries(select(.value.location=="Stockholm"))'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Just try this one as a full copy paste in the shell and you will grasp it.
# pass the multiline string to the jq, use the jq to
# select the attribute named "card_id"
# ONLY if its neighbour attribute
# named "card_id_type" has the "card_id_type-01" value.
# jq -r means give me ONLY the value of the jq query no quotes aka raw
cat << EOF | \
jq -r '.[]| select (.card_id_type == "card_id_type-01")|.card_id'
[
{ "card_id": "id-00", "card_id_type": "card_id_type-00"},
{ "card_id": "id-01", "card_id_type": "card_id_type-01"},
{ "card_id": "id-02", "card_id_type": "card_id_type-02"}
]
EOF
# this ^^^ MUST start first on the line - no whitespace there !!!
# outputs:
# id-01
or with an aws cli command
# list my vpcs or
# list the values of the tags which names are "Name"
aws ec2 describe-vpcs | jq -r '.| .Vpcs[].Tags[]
|select (.Key == "Name") | .Value'|sort -nr
Note that you could move up and down in the hierarchy both during the filtering phase and during the selecting phase :
kubectl get services --all-namespaces -o json | jq -r '
.items[] | select( .metadata.name
| contains("my-srch-string")) |
{ name: .metadata.name, ns: .metadata.namespace
, nodePort: .spec.ports[].nodePort
, port: .spec.ports[].port}
'

bash: parse data from Json file [duplicate]

I have the following json file:
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"name": "Jack",
"location": "Whereever"
}
}
I am using jq and want to get the "name" elements of the objects where 'location' is 'Stockholm'.
I know I can get all names by
cat json | jq .[] | jq ."name"
"Jack"
"Walt"
"Donald"
But I can't figure out how to print only certain objects, given the value of a sub key (here: "location" : "Stockholm").
Adapted from this post on Processing JSON with jq, you can use the select(bool) like this:
$ jq '.[] | select(.location=="Stockholm")' json
{
"location": "Stockholm",
"name": "Walt"
}
{
"location": "Stockholm",
"name": "Donald"
}
To obtain a stream of just the names:
$ jq '.[] | select(.location=="Stockholm") | .name' json
produces:
"Donald"
"Walt"
To obtain a stream of corresponding (key name, "name" attribute) pairs, consider:
$ jq -c 'to_entries[]
| select (.value.location == "Stockholm")
| [.key, .value.name]' json
Output:
["FOO","Donald"]
["BAR","Walt"]
I had a similar related question: What if you wanted the original object format back (with key names, e.g. FOO, BAR)?
Jq provides to_entries and from_entries to convert between objects and key-value pair arrays. That along with map around the select
These functions convert between an object and an array of key-value
pairs. If to_entries is passed an object, then for each k: v entry in
the input, the output array includes {"key": k, "value": v}.
from_entries does the opposite conversion, and with_entries(foo) is a
shorthand for to_entries | map(foo) | from_entries, useful for doing
some operation to all keys and values of an object. from_entries
accepts key, Key, name, Name, value and Value as keys.
jq15 < json 'to_entries | map(select(.value.location=="Stockholm")) | from_entries'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Using the with_entries shorthand, this becomes:
jq15 < json 'with_entries(select(.value.location=="Stockholm"))'
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
}
}
Just try this one as a full copy paste in the shell and you will grasp it.
# pass the multiline string to the jq, use the jq to
# select the attribute named "card_id"
# ONLY if its neighbour attribute
# named "card_id_type" has the "card_id_type-01" value.
# jq -r means give me ONLY the value of the jq query no quotes aka raw
cat << EOF | \
jq -r '.[]| select (.card_id_type == "card_id_type-01")|.card_id'
[
{ "card_id": "id-00", "card_id_type": "card_id_type-00"},
{ "card_id": "id-01", "card_id_type": "card_id_type-01"},
{ "card_id": "id-02", "card_id_type": "card_id_type-02"}
]
EOF
# this ^^^ MUST start first on the line - no whitespace there !!!
# outputs:
# id-01
or with an aws cli command
# list my vpcs or
# list the values of the tags which names are "Name"
aws ec2 describe-vpcs | jq -r '.| .Vpcs[].Tags[]
|select (.Key == "Name") | .Value'|sort -nr
Note that you could move up and down in the hierarchy both during the filtering phase and during the selecting phase :
kubectl get services --all-namespaces -o json | jq -r '
.items[] | select( .metadata.name
| contains("my-srch-string")) |
{ name: .metadata.name, ns: .metadata.namespace
, nodePort: .spec.ports[].nodePort
, port: .spec.ports[].port}
'

Passing JSON objects to jq arguments

Data sample - sample.json (full sample: https://pastebin.com/KFkVmc2M)
{
"ip": 3301234701,
"_shodan": {
"options": {
"referrer": "7ae15507-f5cc-4353-b72e-5cc0b1c34c5e"
},
},
"hash": -1056085507,
"os": null,
"title": "WHM Login",
"opts": {
"vulns": ["!CVE-2014-0160"],
"heartbleed": "2017/08/29 09:57:30 196.196.216.13:2087 - SAFE\
}
},
"isp": "Fiber Grid Inc",
"http": {
"redirects": [],
"title": "WHM Login",
"robots": null,
"favicon": null,
"host": "196.196.216.13",
"html":
}
Script using jq which I hoped would work and haven't found a another solution yet.
cat sample.json | jq \
--arg key0 'Host' \
--arg value0 '.host' \
--arg key1 'Vulnerability' \
--arg value1 '.opts.vulns[0]' \
--arg key2 'ISP' \
--arg value2 '.isp' \
'. | .[$key0]= $value0 | .[$key1]=$value1 | .[$key2]=$value2' \
<<<'{}'
The end result I hoped, but not getting:
{
"Host": "196.196.216.13",
"Vulnerability": "!CVE-2014-0160",
"ISP": "Fiber Grid Inc"
}
Right now it just returns the object as a string and I've tried a lot of different ways to approach the problem. I am quite new to working with JSON and jq but based on what I've read so far, the solution might not be as simple as I'm wishing?
Simply put, why isn't the object being returned as a value of the sample.json object and what do I have to do, to make it work?
Thanks!
to: chepner
{
"196.196.216.13":[
"AS63119",
"Fiber Grid Inc",
"2017-08-29T06:57:22.546423",
"!CVE-2014-0160"
],
"196.196.216.14":[
"AS63119",
"Fiber Grid Inc",
"2017-08-29T06:57:22.546423",
"!CVE-2014-0160"
]
}
Here's a variant solution using the approach suggested by #CharlesDuffy:
$ config='{"Host": ["http", "host"],
"Vulnerability": ["opts", "vulns", 0],
"ISP": ["isp"]}'
$ jq --argjson config "$config" '
. as $in
| $config
| map_values( . as $p | $in | getpath($p) )' pastebin.json
Output:
{
"Host": "196.196.216.13",
"Vulnerability": "!CVE-2014-0160",
"ISP": "Fiber Grid Inc"
}
This incidentally highlights #chepner's point: the care and maintenance required for $config is no less than the care and maintenance required for the corresponding jq query, whereas the jq query language -- by design -- offers far more flexibility.
jq does not support the evaluation of jq expressions in the
way that would be required for your attempt to work. You could do some
kind of shell interpolation, but it would be better to use JSON paths,
e.g. rather than --arg value0 .host, you could write --arg value0 "host", etc. In the following, I've used getpath/1.
There is no need to prepend the jq filter with '. |'
Anyway, assuming the JSON contents of the pastebin are in pastebin.json, you could write:
jq \
--arg key0 Host \
--argjson value0 '["http","host"]' \
--arg key1 Vulnerability \
--argjson value1 '["opts", "vulns", 0]' \
--arg key2 ISP \
--argjson value2 '["isp"]' \
'. as $in
| {}
| .[$key0] = ($in|getpath($value0))
| .[$key1] = ($in|getpath($value1))
| .[$key2] = ($in|getpath($value2))' \
pastebin.json
This would have the result:
{
"Host": "196.196.216.13",
"Vulnerability": "!CVE-2014-0160",
"ISP": "Fiber Grid Inc"
}
In response to the supplementary question ("to: chepner"):
The pastebin data does not contain "196.196.216.14" so the supplementary question is unclear, but you will probably want to start with:
jq '{(.ip_str): [.asn, .isp, .timestamp, opts.vulns[0] ]}' pastebin.json
This produces:
{
"196.196.216.13": [
"AS63119",
"Fiber Grid Inc",
"2017-08-29T06:57:22.546423",
"!CVE-2014-0160"
]
}
The key point here is the pair of parentheses around .ip_str.
As mentioned elsewhere, the jq query is again so simple and straightforward that it seems doubtful that a parameterized version will offer any advantage.

How to capture values from JSON parent and sub-parent parameter using JQ?

I have this below JSON file which I need to capture using JQ, but so far I only manage to capture the parent parameter (SUBSCRIBER_ID), but unable to capture the sub-parent parameter which is "Offer".
Need your guys help on providing a correct JQ filter to capture both "SUBSCRIBER_ID" and "Offer" value.
JSON
{"Data1": [
{"Data2": {
"SUBSCRIBER_ID" : "999050280010099",
"MSISDN" : "999050280010099",
"EMAIL" : "john#email.com",
"OFFERS" : [
{
"Offer" : 12344,
"EffectiveDate" : "1488787236",
"ExpiryDate" : "4070869200"
} ],
"IsGroup" : "false",
}}
]}
My JQ Filter which is not working
'.Data1 | .[] | .Data2 | to_entries | map(.value) | #csv' -r
Expected output:
SUBSCRIBER_ID,Offer
999050280010099,12344
You can try this jq:
jq -r '.Data1|.[]|.Data2|[.SUBSCRIBER_ID, .OFFERS[].Offer]|#csv' file > out.csv
(OR) As suggested by #peak,
jq -r '.Data1[].Data2|[.SUBSCRIBER_ID, .OFFERS[].Offer]|#csv' file
Another one method:
jq -r '.[]|.[]|map([.SUBSCRIBER_ID, .OFFERS[].Offer])|.[]|#csv' file
Input:
$ cat file.json
{
"Data1": [
{
"Data2": {
"SUBSCRIBER_ID": "999050280010099",
"MSISDN": "999050280010099",
"EMAIL": "john#email.com",
"OFFERS": [
{
"Offer": 12344,
"EffectiveDate": "1488787236",
"ExpiryDate": "4070869200"
}
],
"IsGroup": "false"
}
}
]
}
Test:
$ jq -r '.Data1|.[]|.Data2|[.SUBSCRIBER_ID, .OFFERS[].Offer]|#csv' file.json
"999050280010099",12344
$ jq -r '.[]|.[]|map([.SUBSCRIBER_ID, .OFFERS[].Offer])|.[]|#csv' file.json
"999050280010099",12344