I am describing a AWS security group and passing the output by jq in order to get all the CIDRs of the inbound rules.
I have reached so far:
▶ aws ec2 describe-security-groups --group-ids sg-123456789 | jq '.SecurityGroups[0].IpPermissions[0].IpRanges'
[
{
"CidrIp": "11.22.33.44/32",
"Description": "Something"
},
{
"CidrIp": "22.33.44.12/32",
"Description": "Something else"
},
{
"CidrIp": "22.11.33.55/32",
"Description": "Something different"
},
]
I know I can grep but is there a way to get just the CidrIp from each json element of this array?
Sure, change your pipeline to
jq -r '.SecurityGroups[0].IpPermissions[0].IpRanges[].CidrIp'
Demo
Note that I also added the -r flag which makes the output raw text instead of JSON.
jq '.SecurityGroups[0].IpPermissions[0].IpRanges | values[].CidrIp'
this seemed to work as well.
Related
I have a file containing the following structure and unknown number of results:
{
"results": [
[
{
"field": "AccountID",
"value": "5177497"
},
{
"field": "Requests",
"value": "50900"
}
],
[
{
"field": "AccountID",
"value": "pro"
},
{
"field": "Requests",
"value": "251"
}
]
],
"statistics": {
"Matched": 51498,
"Scanned": 8673577,
"ScannedByte": 2.72400814E10
},
"status": "HOLD"
}
{
"results": [
[
{
"field": "AccountID",
"value": "5577497"
},
{
"field": "Requests",
"value": "51900"
}
],
"statistics": {
"Matched": 51498,
"Scanned": 8673577,
"ScannedByte": 2.72400814E10
},
"status": "HOLD"
}
There are multiple such results which are indexed as an array with the results folder. They are not seperated by a comma.
I am trying to just print The "AccountID" sorted by "Requests" in ZSH using jq. I have tried flattening them and using:
jq -r '.results[][0] |.value ' filename
jq -r '.results[][1] |.value ' filename
To get the Account ID and Requests seperately and sorting them. I don't think bash has a dictionary that can be used. The problem lies in the file as the Field and value are not key value pair but are both pairs. Therefore extracting them using the above two lines into seperate arrays and sorting by the second array seems a bit too long. I was wondering if there is a way to combine both the operations.
The other way is to combine it all to a string and sort it in ascending order. Python would probably have the best solution but the code requires to be a zsh or bash script.
Solutions that use sed, jq or any other ZSH supported compilers are welcome. If there is a way to create a dictionary in bash, please do let me know.
The projectd output requirement is just the Account ID vs Request Number.
5577497 has 51900 requests
5177497 has 50900 requests
pro has 251 requests
If you don't mind learning a little jq, it will probably be best to write a small jq program to do what you want.
To get you started, consider the following jq program, which assumes your input is a stream of valid JSON objects with a "results" key similar to your sample:
[inputs | .results[] | map( { (.field) : .value} ) | add]
After making minor changes to your input so that it consists of valid JSON objects, an invocation of jq with the -n option produces an array of AccountID/Requests objects:
[
{
"AccountID": "5177497",
"Requests": "50900"
},
{
"AccountID": "pro",
"Requests": "251"
},
{
"AccountID": "5577497",
"Requests": "51900"
}
]
You could (for example) now use jq's group_by to group these objects by AccountID, and thereby produce the result you want.
jq -S '.results[] | map( { (.field) : .value} ) | add' query-results-aggregate \
| jq -s -c 'group_by(.number_of_requests) | .[]'
This does the trick. Thanks to peak for the guidance.
How to parse / substitute value in jquery.
Json file as below:
{
"09800214851900C3": {
"label": "P7-R1-R16:S2",
"name": "Geist Upgradable rPDU",
"state": "normal",
"order": 0,
"type": "i03",
"snmpInstance": 1,
"lifetimeEnergy": "20155338",
"outlet": {},
"alarm": {
"severity": "",
"state": "none"
},
"layout": {
"0": [
"entity/total0",
"entity/phase0",
"entity/phase1",
"entity/phase2"
]
}
}
}
I want to do like below, But this is not working. Any idea/leads on this will be appreciated.
a=09800214851900C3
jsonfile.json | jq '.${a}.label'
Your current try has the following problems :
jsonfile.json isn't a command, so you can't use it as the first token of a command line. You could cat jsonfile.json | jq ..., but the prefered way to have jq work on a file is to use jq 'command' file
you define a variable a in your shell, but you try to reference it inside a single-quoted string, which prevents the shell from expanding it to its actual value. A shell based solution is to use double-quotes to have the variable expanded, but it's preferable to define the variable in the context of jq itself, using a --arg varname value option
09800214851900C3 isn't considered a "simple, identifier-like key" by jq (because it starts with a digit), so the standard way of accessing the value associated to this key (.key) doesn't work and you need to use ."09800214851900C3" or .["09800214851900C3"] instead
In conclusion I believe you will want to use the following command :
jq --arg a 09800214851900C3 '.[$a].label' jsonfile.json
This is a question about the command line json processor jq. It is not about javascript or jQuery or anything else with js and qs in its name :-)
I have input data like:
{ "id": "person1", "name": "wilma", "age": "quite old"}
{ "id": "person2", "name": "fred"}
{ "id": "person1", "name": "betty", "x": "extra"}
I want output like this:
{
"person1": { "name": "betty", "age": "quite old", "x": "extra" },
"person2": { "name": "fred" }
}
I have tried various things!
e.g.
jq -s '.[] | { (.id) : . }' <data
gives
{ "person1": { "id": "person1", "name": "wilma", "age": "quite old" }}
{ "person2": { "id": "person2", "name": "fred" }}
{ "person1": { "id": "person1", "name": "betty", "x": "extra" }}
Which is sort of there, except it's outputting a stream of objects instead of just one. I need to merge all those objects together.
jqplay.org example
I've also tried using group_by(.id)[]|add which merges each item but still results in a stream. https://jqplay.org/s/lh6QUQ0DO4
You could tweak your attempt as follows:
jq -s 'map({ (.id) : . }) | add' <data
However, it would be more efficient to use inputs and reduce with the -n command-line option instead of -s.
Of course, using this approach runs the risk of collisions.
You might also want to add del(.id)
Ah! I've got it! Or I've got one solution - please post if there's a better way.
jq -s '[group_by(.id)[]| add | { (.id) : . } ]|add' <data
https://jqplay.org/s/BfAdRBZUMW
group_by groups the inputs by their .id value and produces an array of arrays - the inner arrays are the values that match on id.
for each group the inner arrays are passed to add which, because the things in the inner arrays are objects, merges them.
That leaves a 2 item array. We feed that to an object constructor which plucks the id as the key and the whole item as the value. This still leaves an array of items.
the outer [] (starts at start of pattern) says take all those and feed it to add (again), which merges the final objects created in (3).
It works, but there may be a cleaner way.
EDIT
This is uglier but produces the same result and is ~24% faster on a 9MB dataset.
jq -s 'reduce [.[]|{ (.id) : . }][] as $item ({}; . * $item )' <data
This uses reduce <list> as <$var> (<initiation>; <iteration>) starting with an empty object {} and using the merge operator * starting from the incoming item . to create the output. I'm surprised it's faster, but I understand that group_by does a sort, so I guess that's an additional time cost.
Haven't used jq before but I'm wanting to build a shell script that will get a JSON response and extract just the values. To learn I thought I would try on my blog's WP API but for some reason I'm getting an error of:
jq: error (at :322): Cannot index array with string "slug"
When researching for and testing previous questions:
jq: Cannot index array with string
jq is sed for JSON
JSON array to bash variables using jq
How to use jq in a shell pipeline?
How to extract data from a JSON file
The above reading I've tried to code:
URL="http://foobar.com"
RESPONSE=$(curl -so /dev/null -w "%{http_code}" $URL)
WPAPI="/wp-json/wp/v2"
IDENTIFIER="categories"
if (("$RESPONSE" == 200)); then
curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | jq '.' >> $IDENTIFIER.json
result=$(jq .slug $IDENTIFIER.json)
echo $result
else
echo "Not returned status 200";
fi
An additional attempt changing the jq after the curl:
curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | jq '.' | $IDENTIFIER.json
result=(jq -r '.slug' $IDENTIFIER.json)
echo $result
I can modify the uncompress with the python JSON tool:
result=(curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | python -m json.tool > $IDENTIFIER.json)
I can save the JSON to a file but when I use jq I cannot get just the slug and here are my other trys:
catCalled=$(curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | python -m json.tool | ./jq -r '.slug')
echo $catCalled
My end goal is to try to use jq in a shell script and build a slug array with jq. What am I doing wrong in my jq and can I use jq on a string without creating a file?
Return from curl after uncompress per comment request:
[
{
"id": 4,
"count": 18,
"description": "",
"link": "http://foobar.com/category/foo/",
"name": "Foo",
"slug": "foo",
"taxonomy": "category",
},
{
"id": 8,
"count": 9,
"description": "",
"link": "http://foobar.com/category/bar/",
"name": "Bar",
"slug": "bar",
"taxonomy": "category",
},
{
"id": 5,
"count": 1,
"description": "",
"link": "http://foobar.com/category/mon/",
"name": "Mon",
"slug": "mon",
"taxonomy": "category",
},
{
"id": 11,
"count": 8,
"description": "",
"link": "http://foobar.com/category/fort/",
"name": "Fort",
"slug": "fort",
"taxonomy": "category",
}
]
eventually my goal is trying to get the name of the slug's into an array like:
catArray=('foo','bar','mon', 'fort')
There are 2 issues here:
slug is not a root level element in your example json. The root level element is an array. If you want to access the slug property of each element of the array, you can do so like this:
jq '.[].slug' $IDENTIFIER.json
Your example json has trailing commas after the last property of each array element. Remove the commas after "taxonomy": "category".
If I take your sample json, remove the errant commas, save it to a plain text file called test.json and run the following command:
jq '.[].slug' test.json
I get the following output:
"foo"
"bar"
"mon"
"fort"
Preprocessing
Unfortunately, the JSON-like data shown as having been produced by curl is not strictly JSON. jq does not have a "relaxed JSON" mode, so in order to use jq, you will have to preprocess the JSON-like data, e.g. using hjson (see http://hjson.org/):
$ hjson -j input.qjson > input.json
jq
With the JSON in input.json:
$ jq -c 'map(.slug)' input.json
["foo","bar","mon","fort"]
your string is not json, notice how the last member of your objects ends with a comma,
{foo:"bar",baz:9,}
this is legal in javascript, but it's illegal in json. if you are supposed to be receiving json from that endpoint, then contact the people behind it and tell them to fix the bug (it's breaking the json specs by ending objects's last member with a comma, which is illegal in json.) - until it's fixed, i guess you can patch it with a little regex, but it's a dirty quickfix, and probably not very reliable, but running it through
perl -p -0777 -e 's/\"\,\s*}/\"}/g;' makes it legal json..
My json looks like this :
{
"20160522201409-jobsv1-1": {
"vmStateDisplayName": "Ready",
"servers": {
"20160522201409 jobs_v1 1": {
"serverStateDisplayName": "Ready",
"creationDate": "2016-05-22T20:14:22.000+0000",
"state": "READY",
"provisionStatus": "PENDING",
"serverRole": "ROLE",
"serverType": "SERVER",
"serverName": "20160522201409 jobs_v1 1",
"serverId": 2902
}
},
"isAdminNode": true,
"creationDate": "2016-05-22T20:14:23.000+0000",
"totalStorage": 15360,
"shapeId": "ot1",
"state": "READY",
"vmId": 4353,
"hostName": "20160522201409-jobsv1-1",
"label": "20160522201409 jobs_v1 ADMIN_SERVER 1",
"ipAddress": "10.252.159.39",
"publicIpAddress": "10.252.159.39",
"usageType": "ADMIN_SERVER",
"role": "ADMIN_SERVER",
"componentType": "jobs_v1"
}
}
My key keeps changing from time to time. So for example 20160522201409-jobsv1-1 may be something else tomorrow. Also I may more than one such entry in the json payload.
I want to echo $KEYS and I am trying to do it using jq.
Things I have tried :
| jq .KEYS is the command i use frequently.
Is there a jq command to display all the primary keys in the json?
I only care about the hostname field. And I would like to extract that out. I know how to do it using grep but it is NOT a clean approach.
You can simply use: keys:
% jq 'keys' my.json
[
"20160522201409-jobsv1-1"
]
And to get the first:
% jq -r 'keys[0]' my.json
20160522201409-jobsv1-1
-r is for raw output:
--raw-output / -r: With this option, if the filter’s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.
Source
If you want a known value below an unknown property, eg xxx.hostName:
% jq -r '.[].hostName' my.json
20160522201409-jobsv1-1