Data sample - sample.json (full sample: https://pastebin.com/KFkVmc2M)
{
"ip": 3301234701,
"_shodan": {
"options": {
"referrer": "7ae15507-f5cc-4353-b72e-5cc0b1c34c5e"
},
},
"hash": -1056085507,
"os": null,
"title": "WHM Login",
"opts": {
"vulns": ["!CVE-2014-0160"],
"heartbleed": "2017/08/29 09:57:30 196.196.216.13:2087 - SAFE\
}
},
"isp": "Fiber Grid Inc",
"http": {
"redirects": [],
"title": "WHM Login",
"robots": null,
"favicon": null,
"host": "196.196.216.13",
"html":
}
Script using jq which I hoped would work and haven't found a another solution yet.
cat sample.json | jq \
--arg key0 'Host' \
--arg value0 '.host' \
--arg key1 'Vulnerability' \
--arg value1 '.opts.vulns[0]' \
--arg key2 'ISP' \
--arg value2 '.isp' \
'. | .[$key0]= $value0 | .[$key1]=$value1 | .[$key2]=$value2' \
<<<'{}'
The end result I hoped, but not getting:
{
"Host": "196.196.216.13",
"Vulnerability": "!CVE-2014-0160",
"ISP": "Fiber Grid Inc"
}
Right now it just returns the object as a string and I've tried a lot of different ways to approach the problem. I am quite new to working with JSON and jq but based on what I've read so far, the solution might not be as simple as I'm wishing?
Simply put, why isn't the object being returned as a value of the sample.json object and what do I have to do, to make it work?
Thanks!
to: chepner
{
"196.196.216.13":[
"AS63119",
"Fiber Grid Inc",
"2017-08-29T06:57:22.546423",
"!CVE-2014-0160"
],
"196.196.216.14":[
"AS63119",
"Fiber Grid Inc",
"2017-08-29T06:57:22.546423",
"!CVE-2014-0160"
]
}
Here's a variant solution using the approach suggested by #CharlesDuffy:
$ config='{"Host": ["http", "host"],
"Vulnerability": ["opts", "vulns", 0],
"ISP": ["isp"]}'
$ jq --argjson config "$config" '
. as $in
| $config
| map_values( . as $p | $in | getpath($p) )' pastebin.json
Output:
{
"Host": "196.196.216.13",
"Vulnerability": "!CVE-2014-0160",
"ISP": "Fiber Grid Inc"
}
This incidentally highlights #chepner's point: the care and maintenance required for $config is no less than the care and maintenance required for the corresponding jq query, whereas the jq query language -- by design -- offers far more flexibility.
jq does not support the evaluation of jq expressions in the
way that would be required for your attempt to work. You could do some
kind of shell interpolation, but it would be better to use JSON paths,
e.g. rather than --arg value0 .host, you could write --arg value0 "host", etc. In the following, I've used getpath/1.
There is no need to prepend the jq filter with '. |'
Anyway, assuming the JSON contents of the pastebin are in pastebin.json, you could write:
jq \
--arg key0 Host \
--argjson value0 '["http","host"]' \
--arg key1 Vulnerability \
--argjson value1 '["opts", "vulns", 0]' \
--arg key2 ISP \
--argjson value2 '["isp"]' \
'. as $in
| {}
| .[$key0] = ($in|getpath($value0))
| .[$key1] = ($in|getpath($value1))
| .[$key2] = ($in|getpath($value2))' \
pastebin.json
This would have the result:
{
"Host": "196.196.216.13",
"Vulnerability": "!CVE-2014-0160",
"ISP": "Fiber Grid Inc"
}
In response to the supplementary question ("to: chepner"):
The pastebin data does not contain "196.196.216.14" so the supplementary question is unclear, but you will probably want to start with:
jq '{(.ip_str): [.asn, .isp, .timestamp, opts.vulns[0] ]}' pastebin.json
This produces:
{
"196.196.216.13": [
"AS63119",
"Fiber Grid Inc",
"2017-08-29T06:57:22.546423",
"!CVE-2014-0160"
]
}
The key point here is the pair of parentheses around .ip_str.
As mentioned elsewhere, the jq query is again so simple and straightforward that it seems doubtful that a parameterized version will offer any advantage.
Related
json can be created using the following command.
jq -n \
--arg v1 "Value1" \
--arg v2 "Value2" \
'{k1: "$v1", k2:$v2'}
But when my key is mutable, how should I loop? For example, the script I execute is
test.sh k1=v1 k2=v2 k3=v3
test.sh is as follows
index=1
while ((index <= "$#")); do
data_i_key=$(echo ${!index} | awk -F "=" '{print $1}')
data_i_value=$(echo ${!index} | awk -F "=" '{print $2}')
let index++
JSON_STRING=$(jq -n \
--arg value "$data_i_value" \
'{'"$data_i_key"': $value'})
echo $JSON_STRING
The above print result is
{ "K3": "V3" }
if I replace it with
JSON_STRING+=$(jq -n \
--arg val_value "$dataValue" \
'{'"$data_i_key"': $val_value'})
The print result is
{ "k1": "v1" }{ "k2": "v2" }{ "K3": "V3" }
The above two methods have not achieved the goal, do you have a good way to deal with it? My desired output is
{ "k1": "v1" , "k2": "v2" ,"K3": "V3" }
hope you can help me.
I'd suggest a totally different, but simpler, approach:
for kv; do
echo "$kv" | jq -R './"=" | {key:first,value:last}'
done | jq -s 'from_entries'
It builds {key: …, value: …} objects from your positional parameters (splitting them by the equal sign) and then slurping all those objects in a second jq process, converting them into a single object via from_entries.
Alternatively, using -n (--null-input), --args and then accessing via $ARGS.positional. This avoids the loop and the second jq call altogether.
jq -n '$ARGS.positional | map(./"=" | {key:first,value:last}) | from_entries' --args "$#"
If your values can contain = themselves, you have to join all but the first value of the array:
jq -n '$ARGS.positional
| map(./"=" | {key:first,value:.[1:]|join("=")})
| from_entries' --args "$#"
Use parentheses around the key expression to have it evaluated:
jq -n --arg k1 "Key1" --arg v1 "Value1" '{($k1): $v1}'
{
"Key1": "Value1"
}
Perhaps this, which will handle an arbitrary number of key=value arguments:
#!/usr/bin/env bash
args=()
for arg; do
IFS="=" read -r k v <<<"$arg"
args+=("$k" "$v")
done
jq -n --args '
$ARGS.positional as $a
| reduce range(0; $a | length; 2) as $i ({}; .[$a[$i]] = $a[$i + 1])
' "${args[#]}"
Produces:
$ ./test.sh k1=v1 k2=v2 k3=v3
{
"k1": "v1",
"k2": "v2",
"k3": "v3"
}
Data sample - sample.json (full sample: https://pastebin.com/KFkVmc2M)
{
"ip": 3301234701,
"_shodan": {
"options": {
"referrer": "7ae15507-f5cc-4353-b72e-5cc0b1c34c5e"
},
},
"hash": -1056085507,
"os": null,
"title": "WHM Login",
"opts": {
"vulns": ["!CVE-2014-0160"],
"heartbleed": "2017/08/29 09:57:30 196.196.216.13:2087 - SAFE\
}
},
"isp": "Fiber Grid Inc",
"http": {
"redirects": [],
"title": "WHM Login",
"robots": null,
"favicon": null,
"host": "196.196.216.13",
"html":
}
Script using jq which I hoped would work and haven't found a another solution yet.
cat sample.json | jq \
--arg key0 'Host' \
--arg value0 '.host' \
--arg key1 'Vulnerability' \
--arg value1 '.opts.vulns[0]' \
--arg key2 'ISP' \
--arg value2 '.isp' \
'. | .[$key0]= $value0 | .[$key1]=$value1 | .[$key2]=$value2' \
<<<'{}'
The end result I hoped, but not getting:
{
"Host": "196.196.216.13",
"Vulnerability": "!CVE-2014-0160",
"ISP": "Fiber Grid Inc"
}
Right now it just returns the object as a string and I've tried a lot of different ways to approach the problem. I am quite new to working with JSON and jq but based on what I've read so far, the solution might not be as simple as I'm wishing?
Simply put, why isn't the object being returned as a value of the sample.json object and what do I have to do, to make it work?
Thanks!
to: chepner
{
"196.196.216.13":[
"AS63119",
"Fiber Grid Inc",
"2017-08-29T06:57:22.546423",
"!CVE-2014-0160"
],
"196.196.216.14":[
"AS63119",
"Fiber Grid Inc",
"2017-08-29T06:57:22.546423",
"!CVE-2014-0160"
]
}
Here's a variant solution using the approach suggested by #CharlesDuffy:
$ config='{"Host": ["http", "host"],
"Vulnerability": ["opts", "vulns", 0],
"ISP": ["isp"]}'
$ jq --argjson config "$config" '
. as $in
| $config
| map_values( . as $p | $in | getpath($p) )' pastebin.json
Output:
{
"Host": "196.196.216.13",
"Vulnerability": "!CVE-2014-0160",
"ISP": "Fiber Grid Inc"
}
This incidentally highlights #chepner's point: the care and maintenance required for $config is no less than the care and maintenance required for the corresponding jq query, whereas the jq query language -- by design -- offers far more flexibility.
jq does not support the evaluation of jq expressions in the
way that would be required for your attempt to work. You could do some
kind of shell interpolation, but it would be better to use JSON paths,
e.g. rather than --arg value0 .host, you could write --arg value0 "host", etc. In the following, I've used getpath/1.
There is no need to prepend the jq filter with '. |'
Anyway, assuming the JSON contents of the pastebin are in pastebin.json, you could write:
jq \
--arg key0 Host \
--argjson value0 '["http","host"]' \
--arg key1 Vulnerability \
--argjson value1 '["opts", "vulns", 0]' \
--arg key2 ISP \
--argjson value2 '["isp"]' \
'. as $in
| {}
| .[$key0] = ($in|getpath($value0))
| .[$key1] = ($in|getpath($value1))
| .[$key2] = ($in|getpath($value2))' \
pastebin.json
This would have the result:
{
"Host": "196.196.216.13",
"Vulnerability": "!CVE-2014-0160",
"ISP": "Fiber Grid Inc"
}
In response to the supplementary question ("to: chepner"):
The pastebin data does not contain "196.196.216.14" so the supplementary question is unclear, but you will probably want to start with:
jq '{(.ip_str): [.asn, .isp, .timestamp, opts.vulns[0] ]}' pastebin.json
This produces:
{
"196.196.216.13": [
"AS63119",
"Fiber Grid Inc",
"2017-08-29T06:57:22.546423",
"!CVE-2014-0160"
]
}
The key point here is the pair of parentheses around .ip_str.
As mentioned elsewhere, the jq query is again so simple and straightforward that it seems doubtful that a parameterized version will offer any advantage.
I want to create JSON output with jq that looks like this:
{
"records": [
{
"id": "1234",
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
I assumed I have to twiddle with the "filter" of jq whose concept I don't fully get after reading the doc.
This is what I got so far:
$ jq --arg id 1234 \
--arg song Yesterday \
--arg artist "The Beatles" \
'.' \
<<<'{ "records" : [{ "id":"$id", "song":"$song", "artist":"$artist" }] }'
which prints
{
"records": [
{
"id" : "$id",
"song" : "$song",
"artist" : "$artist"
}
]
}
Do I modify the filter? Do I change the input?
An alternate way to your original attempt, on jq-1.6 you can use the $ARGS.positional attribute to construct your JSON from scratch
jq -n '
$ARGS.positional | {
records: [
{
id: .[0],
song: .[1],
artist: .[2]
}
]
}' --args 1234 Yesterday "The Beatles"
As for why your original attempt didn't work, looks you are not modifying your json at all, with your filter '.' you are basically just reading in and printing out "untouched". The arguments set using --arg need to be set to the object inside the filter.
You are looking for something like this:
jq --null-input \
--arg id 1234 \
--arg song Yesterday \
--arg artist "The Beatles" \
'.records[0] = {$id, $song, $artist}'
Each variable reference between curly brackets is converted to a key-value pair where its name is the key, and its value is the value. And assigning the resulting object to .records[0] forces the creation of the surrounding structure.
jq --null-input\
--argjson id 1234\
--arg song Yesterday\
--arg artist "The Beatles"\
'{ "records" : [{ $id, $song, $artist }] }'
gives
{
"records": [
{
"id": 1234,
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
I think you got the JSON/JQ the wrong way round:
This should be your JQ script:
rec.jq
{
records: [
{
id: $id,
song: $song,
artist: $artist
}
]
}
And this should be your JSON (empty):
rec.json
{}
Then:
jq --arg id 123 --arg song "Yesterday" --arg artist "The Beatles" -f rec.jq rec.json
Which produces:
{
"records": [
{
"id": "123",
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
Start with an empty JSON and add the missing bits:
$ jq --arg id 1234 \
--arg song Yesterday \
--arg artist "The Beatles" \
'. | .records[0].id=$id | .records[0].song=$song | .records[0].artist=$artist' \
<<<'{}'
Outputs
{
"records": [
{
"id": "1234",
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
Another, cleaner, approach based on the answer of #Inian could be
jq -n \
--arg id 1234
--arg song Yesterday
--arg artist "The Beatles"
'{records: [{id:$id, song:$song, artist:$artist}]}'
I have the following JSON outp
{
"environment": {
"reg": "abc"
},
"system": {
"svcs": {
"upsvcs": [
{
"name": "monitor",
"tags": [],
"vmnts": [],
"label": "upsvcs",
"credentials": {
"Date": "Feb152018",
"time": "1330"
}
},
{
"name": "application",
"tags": [],
"vmnts": [],
"label": "upsvcs",
"credentials": {
"lastViewed": "2018-02-07"
}
}
]
}
}
and to retrieve Date value (from credentials). I have tried `curl xxx | jq -r '. | select (.Date)'
which is not returning any value. Can someone please let me know what is the correct syntax and any explanation on how to retrieve elements (or any articles that do so).
TIA
The (or at least a) short answer is:
.system.svcs.upsvcs[0].credentials.Date
Without a schema, it is often a bit tricky to get the exact path correctly, so you might want to consider using the jq filter paths. In your case:
$ jq -c paths input.json | grep Date
["system","svcs","upsvcs",0,"credentials","Date"]
You could also use this path (i.e., the above array) directly:
$ jq 'getpath(["system","svcs","upsvcs",0,"credentials","Date"])' input.json
"Feb152018"
and thus you could use a "path-free" query:
$ jq --argjson p $(jq -c paths input.json | grep --max 1 Date) 'getpath($p)' input.json
"Feb152018"
Another approach would be just to retrieve all “truthy” .Date values, no matter where they appear:
$ jq -c '.. | .Date? // empty' input.json
"Feb152018"
select
Since you mentioned select, please note that:
$ jq -c '.. | select(.Date?)' input.json
{"Date":"Feb152018","time":"1330"}
Further information
For further information about jq in general and the topic of retrieval in particular, see the online tutorial, manual, and FAQ:
https://stedolan.github.io/jq/tutorial/
https://stedolan.github.io/jq/manual/v1.5/
https://github.com/stedolan/jq/wiki/FAQ
I'm trying to create a JSON file by executing the following command:
jq --arg greeting world '{"hello":"$greeting"}' > file.json
This command stuck without any input. While
jq -n --arg greeting world '{"hello":"$greeting"}' > file.json
doesn't parse correctly. I'm just wondering is really possible to create a JSON file.
So your code doesn't work because included the variable inside double quotes which gets treated as string. That is why it is not working
As #Jeff Mercado, pointed out the solution is
jq -n --arg greeting world '{"hello":$greeting}' > file.json
About the - in a name. This is actually possible. But as of now this is not available in released version of jq. If you compile the master branch of jq on your system. There is a new variable called $ARGS.named which can be used to access the information.
I just compiled and check the below command and it works like a charm
./jq -n --arg name-is tarun '{"name": $ARGS.named["name-is"]}'
{
"name": "tarun"
}
$ARGS provides access to named (--arg name value) and positional (--args one two three) arguments from the jq command line, and allows you to build up objects easily & safely.
Named arguments:
$ jq -n '{christmas: $ARGS.named}' \
--arg one 'partridge in a "pear" tree' \
--arg two 'turtle doves'
{
"christmas": {
"one": "partridge in a \"pear\" tree",
"two": "turtle doves"
}
}
Positional arguments:
$ jq -n '{numbers: $ARGS.positional}' --args 1 2 3
{
"numbers": [
"1",
"2",
"3"
]
}
Note you can access individual items of the positional array, and that the named arguments are directly available as variables:
jq -n '{first: {name: $one, count: $ARGS.positional[0]}, all: $ARGS}' \
--arg one 'partridge in a "pear" tree' \
--arg two 'turtle doves' \
--args 1 2 3
{
"first": {
"name": "partridge in a \"pear\" tree",
"count": "1"
},
"all": {
"positional": [
"1",
"2",
"3"
],
"named": {
"one": "partridge in a \"pear\" tree",
"two": "turtle doves"
}
}
}
To add to what Jeff and Tarun have already said, you might want to use the \() string interpolation syntax in your command. eg.
jq -n --arg greeting world '{"hello":"\($greeting)"}'
for me this produces
{
"hello": "world"
}
Regarding your reply to Jeff's comment, the argument name you choose has to be a valid jq variable name so an arg like greeting-for-you won't work but you could use underscores so greeting_for_you would be ok. Or you could use the version Tarun described.