I've got an array of multiple JSON. I would like to get the number of of JSON which contain a specific value.
Example:
[
{
"key": "value1",
"2ndKey":"2ndValue1"
},
{
"key": "value2",
"2ndKey":"2ndValue2"
},
{
"key": "value1",
"2ndKey":"2ndValue3"
}
]
So in case I'm looking for value1 in key, the result should be 2.
I would like to get an solution using jq. I had already some tries, however they did not fully work. The best one yet was the following:
cat /tmp/tmp.txt | jq ' select(.[].key == "value1" ) | length '
I get the correct results but it is shown multiple times.
Can anybody help me to further improve my code. Thanks in advance!
You are pretty close. Try this
map(select(.key == "value1")) | length
or the equivalent
[ .[] | select(.key == "value1") ] | length
An efficient and convenient way to count is to use 'count' as defined below:
def count(s; cond): reduce s as $x (0; if ($x|cond) then .+1 else . end);
count(.[]; .key == "value1")
Related
I have the following JSON. From there I'd like to count how many objects I have which type attribute is either "null" or has an array that contains the value "null". In the following example, the answer would be two. Note that the JSON could also be deeply nested.
{
"A": {
"type": "string"
},
"B": {
"type": "null"
},
"C": {
"type": [
"null",
"string"
]
}
}
I came up with the following, but obviously this doesn't work since it misses the arrays. Any hints how to solve this?
jq '[..|select(.type?=="null")] | length'
This answer focuses on efficiency, straightforwardness, and generality.
In brief, the following jq program produces 2 for the given example.
def count(s): reduce s as $x (0; .+1);
def hasValue($value):
has("type") and
(.type | . == $value or (type == "array" and any(. == $value)));
count(.. | objects | select(hasValue("null")))
Notice that using this approach, it would be easy to count the number of objects having null or "null":
count(.. | objects | select(hasValue("null") or hasValue(null)))
You were almost there. For arrays you could use IN. I also used objects, strings and arrays which are shortcuts to a select of the according types.
jq '[.. | objects.type | select(strings == "null", IN(arrays[]; "null"))] | length'
2
Demo
On larger structures you could also improve performance by not creating that array of which you would only calculate the length, but by instead just iterating over the matching items (e.g. using reduce) and counting on the go.
jq 'reduce (.. | objects.type | select(strings == "null", IN(arrays[]; "null"))) as $_ (0; .+1)'
2
Demo
I have json data from curl like this:
[
{
"state": "foo",
},
{
"state": "foo",
},
{
"state": "foo",
}
]
output of curl uri | jq .[].state would be
"foo"
"foo"
"foo"
How can I check with jq if all the "state" values are equal and use this true or false status next in my condition? in this example I would get true value.
but here
"foo"
"foo"
"bar"
should be false
Warning: highly inefficient
Get all .state where value is not foo
Check if length == 0
jq 'map(select(.state != "foo")) | length == 0'
Jq▷Play |~| TRUE
Jq▷Play |~| FALSE
It would be easy to come up with an inefficient solution, e.g. using map or unique, but it's also quite easy to use all/2 to devise an efficient, generic solution.
Consider the following stream-oriented function:
# Return `true` if s is the empty stream
def allEquals(s):
(first(s) // null) as $x | all(s; .==$x);
Using this, we can efficiently test for the equality of all the .state values in an array of objects as in the sample data, while ignoring other keys:
allEquals(.[].state)
Footnote
Of course, if the objects in the array have only one key, as in the sample shown in the Q, one could simply write:
.[0] as $x | all(.[]; . == $x)
Thanks to jq's neat handling of edge cases, this works even if the input array is empty.
this code [.[].state == ("foo") ] | all works as expected
I would like to "transpose" (not sure that's the right word) JSON elements.
For example, I have a JSON file like this:
{
"name": {
"0": "fred",
"1": "barney"
},
"loudness": {
"0": "extreme",
"1": "not so loud"
}
}
... and I would like to generate a JSON array like this:
[
{
"name": "fred",
"loudness": "extreme"
},
{
"name": "barney",
"loudness": "not so loud"
}
]
My original JSON has many more first level elements than just "name" and "loudness", and many more names, features, etc.
For this simple example I could fully specify the transformation like this:
$ echo '{"name":{"0":"fred","1":"barney"},"loudness":{"0":"extreme","1":"not so loud"}}'| \
> jq '[{"name":.name."0", "loudness":.loudness."0"},{"name":.name."1", "loudness":.loudness."1"}]'
[
{
"name": "fred",
"loudness": "extreme"
},
{
"name": "barney",
"loudness": "not so loud"
}
]
... but this isn't feasible for the original JSON.
How can jq create the desired output while being key-agnostic for my much larger JSON file?
Yes, transpose is an appropriate word, as the following makes explicit.
The following generic helper function makes for a simple solution that is completely agnostic about the key names, both of the enclosing object and the inner objects:
# Input: an array of values
def objectify($keys):
. as $in | reduce range(0;length) as $i ({}; .[$keys[$i]] = $in[$i]);
Assuming consistency of the ordering of the inner keys
Assuming the key names in the inner objects are given in a consistent order, a solution can now obtained as follows:
keys_unsorted as $keys
| [.[] | [.[]]] | transpose
| map(objectify($keys))
Without assuming consistency of the ordering of the inner keys
If the ordering of the inner keys cannot be assumed to be consistent, then one approach would be to order them, e.g. using this generic helper function:
def reorder($keys):
. as $in | reduce $keys[] as $k ({}; .[$k] = $in[$k]);
or if you prefer a reduce-free def:
def reorder($keys): [$keys[] as $k | {($k): .[$k]}] | add;
The "main" program above can then be modified as follows:
keys_unsorted as $keys
| (.[$keys[0]]|keys_unsorted) as $inner
| map_values(reorder($inner))
| [.[] | [.[]]] | transpose
| map(objectify($keys))
Caveat
The preceding solution only considers the key names in the first inner object.
Building upon Peak's solution, here is an alternative based on group_by to deal with arbitrary orders of inner keys.
keys_unsorted as $keys
| map(to_entries[])
| group_by(.key)
| map(with_entries(.key = $keys[.key] | .value |= .value))
Using paths is a good idea as pointed out by Hobbs. You could also do something like this :
[ path(.[][]) as $p | { key: $p[0], value: getpath($p), id: $p[1] } ]
| group_by(.id)
| map(from_entries)
This is a bit hairy, but it works:
. as $data |
reduce paths(scalars) as $p (
[];
setpath(
[ $p[1] | tonumber, $p[0] ];
( $data | getpath($p) )
)
)
First, capture the top level as $data because . is about to get a new value in the reduce block.
Then, call paths(scalars) which gives a key path to all of the leaf nodes in the input. e.g. for your sample it would give ["name", "0"] then ["name", "1"], then ["loudness", "0"], then ["loudness", "1"].
Run a reduce on each of those paths, starting the reduction with an empty array.
For each path, construct a new path, in the opposite order, with numbers-in-strings turned into real numbers that can be used as array indices, e.g. ["name", "0"] becomes [0, "name"].
Then use getpath to get the value at the old path in $data and setpath to set a value at the new path in . and return it as the next . for the reduce.
At the end, the result will be
[
{
"name": "fred",
"loudness": "extreme"
},
{
"name": "barney",
"loudness": "not so loud"
}
]
If your real data structure might be two levels deep then you would need to replace [ $p[1] | tonumber, $p[0] ] with a more appropriate expression to transform the path. Or maybe some of your "values" are objects/arrays that you want to leave alone, in which case you probably need to replace paths(scalars) with something like paths | select(length == 2).
I have JSON data in the following form:
{
"home": [{
"pageid": "about yft",
"pageData": "0908782"
},
{
"pageData": "09897"
}]}
How do I get the total number of home->pageid items?
Here is what I have tried so far:
$curFileName ={{Path_of_json_file.json}}
appName0=$(cat $curFileName | jq -c '.["home"][]["pageid"]' | sed 's/"//g');
${#appName0[#]} # to get the length of pageid but didn't success..
But it didn't return the desired results.
Collect the pageid items of "string" type into an array, then return the length of the array:
n=$(jq '[.home[].pageid | select(type == "string")] | length' < file.json)
Alternatively, check if the items in the home array have pageid. If the item has the property, put 1 into the result array, otherwise put zero. Finally sum the zeroes and ones with add function:
n=$(jq '[.home[] | if has("pageid") then 1 else 0 end] | add' < file.json)
Here is how you should print the number:
printf '%d\n' "$n"
Sample Output
3
The input as originally given by the OP was invalid, and also didn't include an object without .pageid. In this response, the following JSON will be used:
{
"home": [
{
"pageid": "about yft",
"pageData": "0908782"
},
{
"pageData": "09897"
}
]
}
Consider now these two filters, both of which yield 1 for the above JSON:
[.home[] | select(has("pageid")) ] | length
[.home[] | .pageid//empty ] | length
If .home was huge, then a more efficient approach would be as follows:
def count(s): reduce s as $i(0; .+1);
count(.home[] | select(has("pageid")))
And here's an efficient, one-line variant:
reduce (.home[].pageid?//empty) as $x (0; .+1)
Counting the number of distinct items
To obtain the number of distinct home->pageid items, the simplest would be to use unique|length rather than length, e.g. in either of the first two solutions above.
After doing lots of search get the best result which is given below
echo `cat $curFileName | jq '.["home"][]["pageid"]' | wc -l `
Where is:
$curFileName = {{Path_of_json_file.json}}
.["home"][]["pageid"] = target Json Object
json_file.json :
{
"home": [
{
"pageid": "about yft",
"pageData": "0908782"
},
{
"pageData": "09897"
}
]
}
for more
http://www.compciv.org/recipes/cli/jq-for-parsing-json/
I've already asked a similar question here - Parse json and choose items\keys upon condition, but this time it's slightly different.
This is the Example:
[
{
"item1": "value123",
"array0": [
{
"item2": "aaa"
}
]
},
{
"item1": "value456",
"array1": [
{
"item2": "bbb"
}
]
},
{
"item1": "value789",
"array2": [
{
"item2": "ccc"
}
]
}
]
I'd like to get the value of "item1", only when "item2" has a specific value.
Let's say if item2 equals "bbb", then all I want to get back is "value456".
I've tried to solve it with jq like it worked for me in the issue mentioned above, but to no avail, as I can't extract values from a "higher" level than the one i'm searching in with jq's select and map.
An easier solution is available thanks to the magic powers of the recurse operator, ..:
jq -r '.[] | select(.. | .item2? == "bbb").item1'
Basically what this does is, for each (.[]) object in the original array, pick (select) only those in which any of the keys recursively (..) named .item2 equals "bbb", and then select the .item1 property of said object.
found the solution on jq's manual (with #ifthenelse and #Objects) - http://stedolan.github.io/jq/manual
jq -r '{name1: .[].item1, name2: .[].array[].item2} | /
if .item2 == "bbb" then .name1 elif .item2 != "bbb" then empty else empty end'
If you find the magic of .. too powerful for your purposes, the following may be useful. It restricts the search for the "item2" key.
def some(condition): reduce .[] as $x
(false; if . then . else $x|condition end);
.[]
| to_entries
| select( some( .value | (type == "array") and some(.item2 == "bbb")) )
| from_entries
| .item1
If your jq has any/1, then use it instead of some/1.
Here is a solution which uses tostream and getpath
foreach (tostream|select(length==2)) as [$p,$v] (.;.;
if $p[-1] == "item2" and $v == "bbb" then getpath($p[:-3]).item1 else empty end
)
EDIT: I now realize a filter of the form foreach E as $X (.; .; R) can almost always be rewritten as E as $X | R so the above is really just
(tostream | select(length==2)) as [$p,$v]
| if $p[-1] == "item2" and $v == "bbb" then getpath($p[:-3]).item1 else empty end