I would like to write a simple jq file that allows me to count items grouped by a specified key.
I expect the script contents to be something similar too:
group_by($group) | map({group: $group, cnt: length})
and to invoke it something like
cat my.json | jq --from-file count_by.jq --args group .header.messageType
Whatever I've tried the argument always ends up as a string and is not usable as a key.
Since you have not followed the minimal complete verifiable example
guidelines, it's a bit difficult to know what the best approach to your problem will be, but whatever approach you take, it is important to bear in mind that --arg always passes in a JSON string. It cannot be used to pass in a jq program fragment unless the fragment is a JSON string.
So let's consider one option: passing in a JSON object representing a path that you can use in your program.
So the invocation could be:
jq -f count_by.jq --argjson group '["header", "messageType"]'
and the program would begin with:
group_by(getpath($group)) | ...
Having your cake ...
If you really want to pass in arguments such as .header.messageType, there is a way: convert the string $group into a jq path:
($group|split(".")|map(select(length>0))) as $path
So your jq filter would look like this:
($group|split(".")|map(select(length>0))) as $path
| group_by(getpath($path)) | map({group: $group, cnt: length})
Shell string interpolation
If you want a quick bash solution that comes with many caveats:
group=".header.messageType"
jq 'group_by('"$group"') | map({group: "'"$group"'", cnt: length}'
Related
I have a json file locally called pokemini.json. These are the contents of it;
{"name":"Bulbasaur","type":["Grass","Poison"],"total":318,"hp":45,"attack":49}
{"name":"Ivysaur","type":["Grass","Poison"],"total":405,"hp":60,"attack":62}
{"name":"Venusaur","type":["Grass","Poison"],"total":525,"hp":80,"attack":82}
{"name":"VenusaurMega Venusaur","type":["Grass","Poison"],"total":625,"hp":80,"attack":100}
{"name":"Charmander","type":["Fire"],"total":309,"hp":39,"attack":52}
{"name":"Charmeleon","type":["Fire"],"total":405,"hp":58,"attack":64}
{"name":"Charizard","type":["Fire","Flying"],"total":534,"hp":78,"attack":84}
{"name":"CharizardMega Charizard X","type":["Fire","Dragon"],"total":634,"hp":78,"attack":130}
{"name":"CharizardMega Charizard Y","type":["Fire","Flying"],"total":634,"hp":78,"attack":104}
{"name":"Squirtle","type":["Water"],"total":314,"hp":44,"attack":48}
There are a few types of pokemon in here and I want to do some aggregation with jq.
I could, per example, write this command;
> jq -s -c 'group_by(.type[0]) | .[]' pokemini.json
[{"name":"Charmander","type":["Fire"],"total":309,"hp":39,"attack":52},{"name":"Charmeleon","type":["Fire"],"total":405,"hp":58,"attack":64},{"name":"Charizard","type":["Fire","Flying"],"total":534,"hp":78,"attack":84},{"name":"CharizardMega Charizard X","type":["Fire","Dragon"],"total":634,"hp":78,"attack":130},{"name":"CharizardMega Charizard Y","type":["Fire","Flying"],"total":634,"hp":78,"attack":104}]
[{"name":"Bulbasaur","type":["Grass","Poison"],"total":318,"hp":45,"attack":49},{"name":"Ivysaur","type":["Grass","Poison"],"total":405,"hp":60,"attack":62},{"name":"Venusaur","type":["Grass","Poison"],"total":525,"hp":80,"attack":82},{"name":"VenusaurMega Venusaur","type":["Grass","Poison"],"total":625,"hp":80,"attack":100}]
[{"name":"Squirtle","type":["Water"],"total":314,"hp":44,"attack":48}]
I am aware that the -c flag is what is causing it to print line by line and that I need -s to handle the fact that my json file is more like jsonlines that actualy json. It should also be pointed that out there are only three types of pokemon detected because I can grouping over .type[0] (note that [0]).
I don't get why this does not work though;
> jq -s '.[] | group_by(.type[0])' pokemini.json
jq: error (at pokemini.json:10): Cannot index string with string "type"
group_by/1 expects its input to be an array. By calling .[] first, you are effectively undoing the work of the -s option.
By the way, an alternative to using -s is to use inputs with the -n command-line option, but in this case it makes little difference. When you don’t actually need to read all the entire stream of inputs at once, though, using inputs is in general more efficient.
I have a JSON file and I am extracting data from it using jq. One simple use case is pulling out any JSON Object that contains an Id which is provided as an argument.
I use the following simple script to do so:
[.[] | select(.id == $ID)]
The script is stored in a separate file (by_id.jq) which I pass in using the -f argument.
The full command looks something like this:
cat ./my_json_file.json | jq -sf --arg ID "8df993c1-57d5-46b3-a8a3-d95066934e5b" ./by_id.jq
Is there a way by only using jq that a comma separated list of values could be passed as an argument to the jq script and iterate through the ids and check them against the value of .id in the the JSON file with the result being the objects that have that id?
For example if I wanted to pull out three objects by their ids I would want to structure the command in this way:
cat ./my_json_file.json | jq -sf --arg ID "8df993c1-57d5-46b3-a8a3-d95066934e5b,1d5441ca-5758-474d-a9fc-40d0f68aa538,23cc618a-8ad4-4141-bc1c-0251y0663963" ./by_id.jq
Sure. Though you'll need to parse (split) that list of ids to something that jq can work with, such as an array of ids. Then your problem becomes, given an array of keys, select objects that have any of these ids. Which you could use approaches found here.
$ jq --arg ID '8df993c1-57d5-46b3-a8a3-d95066934e5b,1d5441ca-5758-474d-a9fc-40d0f68aa538,23cc618a-8ad4-4141-bc1c-0251y0663963' '
select(.id | IN($ID|split(",")[]))
' ./my_json_file.json
I'm not sure what your input looks like but judging by your use of slurping then filtering the slurped input, it's a stream of objects. The slurping is not necessary here.
Here is an approach that focuses on efficiency.
Your Q indicates that in fact you have a stream of objects, so the first step towards efficiency is to avoid the -s option, and use -n with inputs instead.
The second step it to avoid splitting your comma-separated string of values more than once.
So your script might look like this:
INDEX($ids | splits(","); .) as $dict
| inputs
| select($dict[.id])
And the invocation would look like this:
jq -n --args a,b,c -f by_id.jq
This of course assumes that simply splitting the string of ids on "," will suffice. You might need to trim the values and take care of other potential anomalies.
For efficiency, it would be better to split $ID just once.
So if you have to use the -s option, you could use the following jq program:
INDEX($ID | splits(","); .) as $dict
| .[]
| select($dict[.id])
I have a JSON document that looks like:
simple: 42
normal:
description: "NORMAL"
combo:
one:
description: "ONE"
two:
description: "TWO"
arbitrary:
foo: 42
I want to use a jq expression to generate the following:
["normal", "one", "two"]
The condition to select the key is that its corresponding value is an object type that has a key description. In this case, keys simple and arbitrary don't qualify.
I'm having a hard time to craft the filter. Looked into with_entries and recurse/2 but can't solve it myself.
TIA.
It's not clear to me whether the YAML that you gave is just a "view" of your JSON or whether you actually want to start with YAML. If your document really is YAML, then one approach would be to use a tool
(such as yaml2json or yq) to convert the yaml to JSON, and then run jq
as shown below; another would be to use jq as a text-processor,
but in that case you could just as well use awk.
yaml2json input.yaml |
jq -c '[.. | objects | to_entries[]
| select(.value | has("description")?) | .key]'
Output
["normal","one","two"]
Streaming parser
This type of problem is also well-suited to jq's streaming parser, which is especially handy when dealing with very large JSON texts. Using jq --stream, a suitable jq filter would be:
[select(length==2) | .[0] | select(.[-1] == "description") | .[-2]]
The ordering of the results will depend on the ordering of the keys produced by the YAML-to-JSON conversion tool.
I'm trying to use jq to get a value from the JSON that cURL returns.
This is the JSON cURL passes to jq (and, FTR, I want jq to return "VALUE-I-WANT" without the quotation marks):
[
{
"success":{
"username":"VALUE-I-WANT"
}
}
]
I initially tried this:
jq ' . | .success | .username'
and got
jq: error (at <stdin>:0): Cannot index array with string "success"
I then tried a bunch of variations, with no luck.
With a bunch of searching the web, I found this SE entry, and thought it might have been my saviour (spoiler, it wasn't). But it led me to try these:
jq -r '.[].success.username'
jq -r '.[].success'
They didn't return an error, they returned "null". Which may or may not be an improvement.
Can anybody tell me what I'm doing wrong here? And why it's wrong?
You need to pipe the output of .[] into the next filter.
jq -r '.[] | .success.username' tmp.json
tl;dr
# Extract .success.username from ALL array elements.
# .[] enumerates all array elements
# -r produces raw (unquoted) output
jq -r '.[].success.username' file.json
# Extract .success.username only from the 1st array element.
jq -r '.[0].success.username' file.json
Your input is an array, so in order to access its elements you need .[], the array/object-value iterator (as the name suggests, it can also enumerate the properties of an object):
Just . | sends the input (.) array as a whole through the pipeline, and an array only has numerical indices, so the attempt to index (access) it with .success.username fails.
Thus, simply replacing . | with .[] | in your original attempt, combined with -r to get raw (unquoted output), should solve your problem, as shown in chepner's helpful answer.
However, peak points out that since at least jq 1.3 (current as of this writing is jq 1.5) you don't strictly need a pipeline, as demonstrated in the commands at the top.
So the 2nd command in your question should work with your sample input, unless you're using an older version.
I have to curl to a site (statuscake.com) that sends multiple items back in a JSON, each line of which contains multiple items. I want to extract from each line two of them, WebsiteName and TestID, so I can check if WebsiteName matches the one I'm interested in, get the TestID out and pass this to a second curl statement to delete the test.
Although it's more complex, the JSON that comes back is essentially of the form
[{"TestID": 123, "WebsiteName": "SomeSite1"}, {"TestID": 1234, "WebsiteName": "SomeSite2"}]
I can't seem to find a magic jq command to do it all in one - if there is one, I'd be really happy to see it.
I've got
cat $data | jq '[.[] | .WebsiteName]'
to get an array of the website names (and a very similar one for the TestIDs, but I think I've done something daft. data is the information coming back from the curl to get the JSON and that's populated OK.
I want to be able to assign these to two arrays, names and ids, then search names for the index of the relevant name, grab the id from ids and pass that to the curl. Unless there's a better way.
Any advice please?
My Xidel can do it all at once by selecting the JSON with a XPath-like query:
E.g. return all ids where the WebsiteName contains "site2" from an array of objects:
xidel /tmp/x.json -e '$json()[contains((.).WebsiteName, "site2")]/TestID'
Or e.g. to download the original JSON and then make the HTTP request with the ids:
xidel http://statuscake.com/your-url... -f '$json()[contains((.).WebsiteName, "site2")]/TestID!x"/your-delete-url{.}..."'
If I'm getting your question right, it sounds like what you want is to, for each element, select those where .WebsiteName == "needle", and then get .TestID from it. You can do just that:
.[] | select(.WebsiteName == "needle") | .TestID
If you want an array as the result, you can wrap the above script in square brackets.
The jq filters startswith and endswith may be of interest to you. If you're going to pass the result back to cURL, you may also be interested in the #sh formatting filter and the -r command-line flag.
Assuming you have a bash 4+ and assuming the json is valid (does not contain newlines in strings, etc.) this works:
$ echo "$data"
[{"TestID": 123, "WebsiteName": "SomeSite1"}, {"TestID": 1234, "WebsiteName":
"SomeSite2"}, {"TestID": 555, "WebsiteName": "foo*ba#r blah[54]quux{4,5,6}"}]
$ declare -A arr
$ while IFS= read -r line; do
eval "$line"
done < <(jq -M -r '.[] | #sh "arr[\(.WebsiteName)]+=\(.TestID)"' <<<"$data")
$ declare -p arr
declare -A arr='(["foo*ba#r blah[54]quux{4,5,6}"]="555" [SomeSite2]="1234" [SomeSite1]="123" )'
Here is a solution using only jq primitives.
.[]
| if .WebsiteName == "SomeSite1" then .TestID else empty end
This is essentially the same as Santiago's answer but if you are new to jq it may be informative because select/1 is defined as
def select(f): if f then . else empty end;