This works to search for tactics that equal "impact". However, it will only pull the objects themselves.
jq '.techniques[] | select(.tactic == "impact")'
Is there no way to use select while walking through json with something like jq '. | select(.techniques[].tactic == "impact")'? I'm guessing the issue is that something like that, even if it works, still does not explicitly say to leave the previous items as well.
It is not viable to manually rebuild the parent.
input
{
"viewMode": 0,
"hideDisabled": false,
"techniques": [
{
"name": "john",
"tactic": "reconnaissance"
},
{
"name": "jane",
"tactic": "impact"
},
{
"name": "jill",
"tactic": "execution"
}
],
"karma": "yes"
}
desired output
{
"viewMode": 0,
"hideDisabled": false,
"techniques": [
{
"name": "jane",
"tactic": "impact"
}
],
"karma": "yes"
}
If this is so remedial that it warrants no response, I'll figure it out and update. It seems like the most basic thing. I'll also be doing a !=, which also works fine normally, but doesn't capture the entire body.
I have tried using variables to do it, which get me close;
jq '{techniques: [.techniques[] | select(.tactic == "impact")]} as $a| $a' test.json
However, trying to add a key "techniques" to that array ruins my ability to use it;
jq '{techniques: [.techniques[] | select(.tactic == "impact")]} as $a| $a + [.]' test.json
jq: error (at test.json:19): object ({"technique...) and array ([{"viewMode...) cannot be added
|= is your friend, e.g.
.techniques |= map(select(.tactic == "impact"))
Related
I think I'm a step off from figuring out how to jq reduce via filter a key to another objects sub-key.
I'm trying to combine files (simplified from Elasticsearch's ILM Explain & ILM Policy API responses):
$ echo '{".siem-signals-default": {"modified_date": "siem", "version": 1 }, "kibana-event-log-policy": {"modified_date": "kibana", "version": 1 } }' > ip1.json
$ echo '{"indices": {".siem-signals-default-000001": {"action": "complete", "index": ".siem-signals-default-000001", "policy" : ".siem-signals-default"} } }' > ie1.json
Such that the resulting JSON is:
{
".siem-signals-default-000001": {
"modified_date": "siem",
"version": 1
"action": "complete",
"index": ".siem-signals-default-000001",
"policy": ".siem-signals-default"
}
}
Where ie1 is base JSON and for a child-object, its sub-element policy should line up to ip1's key and copy its sub-elements into itself. I've been trying to build off this, this, and this (from StackOverflow, also this, this, this from external sources). I'll list various rabbit hole attempts building off these, but they're all insufficient:
$ ((cat ie1.json | jq '.indices') && cat ip1.json) | jq -s 'map(to_entries)|flatten|from_entries' | jq '. as $v| reduce keys[] as $k({}; if true then .[$k] += $v[$k] else . end)'
{
".siem-signals-default": {
"modified_date": "siem",
"version": 1
},
".siem-signals-default-000001": {
"action": "complete",
"index": ".siem-signals-default-000001",
"policy": ".siem-signals-default"
},
"kibana-event-log-policy": {
"modified_date": "kibana",
"version": 1
}
}
$ jq --slurpfile ip1 ip1.json '.indices as $ie1|$ie1+{ilm: $ip1 }' ie1.json
{
".siem-signals-default-000001": {
"action": "complete",
"index": ".siem-signals-default-000001",
"policy": ".siem-signals-default"
},
"ilm": [
{
".siem-signals-default": {
"modified_date": "siem",
"version": 1
},
"kibana-event-log-policy": {
"modified_date": "kibana",
"version": 1
}
}
]
}
I also expected something like this to work, but it compile errors
$ jq -s ip1 ip1.json '. as $ie1|$ie1 + {ilm:(keys[] as $k; $ip1 | select(.policy == $ie1[$k]) | $ie1[$k] )}' ie1.json
jq: error: ip1/0 is not defined at <top-level>, line 1:
ip1
jq: 1 compile error
From this you can see, I've determined various ways to join the separate files, but though I have code I thought would play into filtering, it's not correct / taking effect. Does anyone have an idea how to get the filter part working? TIA
This assumes you are trying to combine the .indices object stored in ie1.json with an object within the object stored in ip1.json. As the keys upon to match are different, I further assumed that you want to match the field name from the .indices object, reduced by cutting off everything that comes after the last dash -, to the same key in the object from ip1.json.
To this end, ip1.json is read in from input as $ip (alternatively you can use jq --argfile ip ip1.json for that), then the .indices object is taken from the first input ie1.json and to the inner object accessed via with_entries(.value …) is added the result of a lookup within $ip at the matching and accordingly reduced .key.
jq '
input as $ip | .indices | with_entries(.value += $ip[.key | sub("-[^-]*$";"")])
' ie1.json ip1.json
{
".siem-signals-default-000001": {
"action": "complete",
"index": ".siem-signals-default-000001",
"policy": ".siem-signals-default",
"modified_date": "siem",
"version": 1
}
}
Demo
If instead of the .indices object's inner field nane you want to have the content of field .index as reference (which in your sample data has the same value), you can go with map_values instead of with_entries as you don't need the field's name anymore.
jq '
input as $ip | .indices | map_values(. += $ip[.index | sub("-[^-]*$";"")])
'ie1.json ip1.json
Demo
Note: I used sub with a regex to manipulate the key name, which you can easily adjust to your liking if in reality it is more complicated. If, however, the pattern is infact as simple as cutting off after the last dash, then using .[:rindex("-")] instead will also get the job done.
I also received offline feedback of a simple "workable for my use case" but not exact answer:
$ jq '.indices | map(. * input[.policy])' ie1.json ip1.json
[
{
"action": "complete",
"index": ".siem-signals-default-000001",
"policy": ".siem-signals-default",
"modified_date": "siem",
"version": 1
}
]
Posting in case someone runs into similar, but other answer's better.
I have some JSON like this:
{
"x": [
{
"name": "Hello",
"id": "211"
},
{
"name": "Goodbye",
"id": "221"
},
{
"name": "Christmas",
"id": "171"
}
],
"y": "value"
}
Using jq, given a name value (e.g. Christmas) how can I get it's associated id (i.e. 171).
I've got as far as being able to check for presence of the name in one of the array's objects, but I can't work out how to filter it down
jq -r 'select(.x[].name == "Christmas")'
jq approach:
jq -r '.x[] | select(.name == "Christmas").id' file
171
The function select(boolean_expression) produces its input unchanged if boolean_expression returns true for that input, and produces no output otherwise.
It can also been done like:
jq '.x[] | select(.name == "Christmas").id'
Also you can try this at link online jq play
I'm looking for efficient means to search through an large JSON object for "sub-objects" that match a filter (via select(), I imagine). However, the top-level JSON is an object with arbitrary nesting contained within, including more simple values, objects and arrays of objects. For example:
{
"name": "foo",
"class": "system",
"description": "top-level-thing",
"configuration": {
"status": "normal",
"uuid": "id"
},
"children": [
{
"id": "c1",
"class": "c1",
"children": [
{
"id": "c1.1",
"class": "c1.1"
},
{
"id": "c1.1",
"class": "FINDME"
}
]
},
{
"id": "c2",
"class": "FINDME"
}
],
"thing": {
"id": "c3",
"class": "FINDME"
}
}
I have a solution which does part of what I want (and is understandable):
jq -r '.. | arrays | .[] | select(.class=="FINDME"?) | .id'
which returns:
c2
c1.1
... however, it misses c3, plus it changes the order of items output. Additionally I'm expecting this to operate on potentially very large JSON structures, I would like to make sure I find an efficient solution. Bonus points for something that remains readable by jq neophytes (myself included).
FWIW, references I was using to help me on the way, in case they help others:
Select objects based on value of variable in object using jq
How to use jq to find all paths to a certain key
Recursive search values by key
For small to modest-sized JSON input, you're on the right track with ..
but it seems you want to select objects, like so:
.. | objects | select(.class=="FINDME"?) | .id
For JSON documents that are very large, this might require too much memory, so it may be worth knowing about jq's streaming parser. Unfortunately it's much more difficult to use, so I'd suggest trying the above, and if you're interested, look in the usual places for documentation about the --stream option.
Here's a streaming-parser solution. To make sense of it, you'll need to read up on the --stream option, but the key is that the output includes lines of the form: [PATH, VALUE]
program.jq
foreach inputs as $in (null;
if has("id") and has("class") then null
else . as $x
| $in
| if length != 2 then null
elif .[0][-1] == "id" then ($x + {id: .[-1]})
elif .[0][-1] == "class"
and .[-1] == "FINDME" then ($x + {class: .[-1]})
else $x
end
end;
select(has("id") and has("class")) | .id )
Invocation
jq -n --stream -f program.jq input.json
Output with sample input
"c1.1"
"c2"
"c3"
The structure I ultimately want would be:
{
"catalog": [
{
"name": "X",
"catalog": [
{ "name": "Y", "uniqueId": "Z" },
{ "name": "Q", "uniqueId": "B" }
]
}
]
}
This is what the existing structure looks like except there are many other properties at each level (https://gist.github.com/ajcrites/e0e0ca4ca3a08ff2dc401ec872e6094c). I just want to filter those out and get a JSON format that looks specifically like this.
I have started out with: jq '.catalog', but this returns only the array. I still want the catalog property name there. I can do this with jq '{catalog: .catalog[]}, but this prints out each catalog object individually which makes the whole output invalid JSON. I still want the properties to be in the array. Is there a way to filter specific property key-values within arrays using jq?
The following transforms the given input to the desired output and may well be what you want:
{catalog}
| .catalog |= map( {name, catalog} )
| .catalog[].catalog |= map( {name, uniqueId} )
| .catalog |= .[0:1]
However, it's not clear to me that this is really what you want, as you don't discuss the duplication in the given JSON input. So maybe you don't really want the last line in the above, or maybe you want duplicates to be handled in some other way, or ....
Anyway, the trick to keeping things simple here is to use |=.
An alternative approach would be to use del to delete the unwanted properties (rather than selecting the ones you want), but in the present case, that would be (at best) tedious.
You could start by using tostream to convert your sample.json
into a stream of [path, value] arrays as you can see by running
jq -c tostream sample.json
This will generate
[["catalog",0,"catalog",0,"name"],"Y"]
[["catalog",0,"catalog",0,"prop11"],""]
[["catalog",0,"catalog",0,"uniqueId"],"Z"]
[["catalog",0,"catalog",0,"uniqueId"]]
[["catalog",0,"catalog",1,"name"],"Y"]
[["catalog",0,"catalog",1,"prop11"],""]
...
reduce and setpath can be used to convert back into the
original form with a filter such as:
reduce (tostream|select(length==2)) as [$p,$v] (
{};
setpath($p;$v)
)
Adding conditionals makes it easy to omit properties at any level.
For example the following removes leaf attributes starting with "prop":
reduce (tostream|select(length==2)) as [$p,$v] (
{};
if $p[-1]|startswith("prop")
then .
else setpath($p;$v)
end
)
With your sample.json this produces
{
"catalog": [
{
"catalog": [
{
"name": "Y",
"uniqueId": "Z"
},
{
"name": "Y",
"uniqueId": "Z"
}
],
"name": "X"
},
{
"catalog": [
{
"name": "Y",
"uniqueId": "Z"
},
{
"name": "Y",
"uniqueId": "Z"
}
],
"name": "X"
}
]
}
If the goal is to remove certain properties, then one could do so using walk/1. For example, to remove properties whose names start with "prop":
walk(if type == "object"
then with_entries(select(.key|startswith("prop") | not))
else . end)
The same approach would also be applicable if the focus is on retaining certain properties, e.g.:
walk(if type == "object"
then with_entries(select(.key == "name" or .key == "uniqueId" or .key == "catalog"))
else . end)
You could build up a file that contains paths into the json (expressed as arrays) that you want to keep. Then filter out values that do not fit in those paths.
paths.json:
["catalog","name"]
["catalog","catalog","name"]
["catalog","catalog","uniqueId"]
Then filter values based on their paths. Using streams is a great way to go for this since it gives you access to these paths directly:
$ jq --slurpfile paths paths.json '
def keep_path($path): any($paths[]; . == [$path[] | select(strings)]);
fromstream(tostream | select(length == 1 or keep_path(.[0])))
' input.json
With the help of jq i would like to select all addresses of nodes that have at least one required=true in their attribute list. The result list should have unique items.
Here is my Json:
{
"nodes": [
{
"address":"127.0.0.1",
"attributes": [
{
"id":"abc",
"required":true
},
{
"id":"def",
"required":true
},
{
"id":"ghi",
"required":false
}
]
},
{
"address":"127.0.0.2",
"attributes": [
{
"id":"abc",
"required":false
},
{
"id":"def",
"required":false
}
]
}
]
}
I first tried with:
jq '.nodes[] | select(.attributes[].required == true) | .address'
This produces:
"127.0.0.1"
"127.0.0.1"
So it gets the address for every required=true field it finds in the attributes section. How to make the result list unique? There is also a unique keyword in jq, but I couldn't figure out how this could help me.
Using unique is safe but it does require sorting, which may not be necessary. In your particular case, for example, the repetition is an artifact of the jq query. Consider using any instead (or as well), as it more precisely captures the intention ("at least one"), as well as having "short-circuit" semantics (i.e., it stops searching once the condition is true):
$ jq '.nodes[]
| select( any(.attributes[]; .required == true))
| .address' input.json
Output:
"127.0.0.1"
You can always add unique if necessary.