identify an item in json and update key and value in it - json

my input json is given below , i want select if "Name": "bot-block" then update it's ​"Action": to "Block": {} from "Allow": {},i performed this this using select command but it filters my json and only returns the item with .Name=bot-block,i want updation in json not filtering .this is my current command jq '.[] | select(.Name=="bot-block") | .Action |= . + { "Block" : {} } ' input.json
[
{
"Name": "searchblock",
"Priority": 3,
"Action": {
"Block": {}
},
"VisibilityConfig": {
"SampledRequestsEnabled": true,
"CloudWatchMetricsEnabled": true,
"MetricName": "searchblock"
}
},
{
"Name": "bot-block",
"Priority": 4,
"Action": {
"Allow": {}
},
"VisibilityConfig": {
"SampledRequestsEnabled": true,
"CloudWatchMetricsEnabled": true,
"MetricName": "user-agent"
}
}
]
expected output
[
{
"Name": "searchblock",
"Priority": 3,
"Action": {
"Block": {}
},
"VisibilityConfig": {
"SampledRequestsEnabled": true,
"CloudWatchMetricsEnabled": true,
"MetricName": "searchblock"
}
},
{
"Name": "bot-block",
"Priority": 4,
"Action": {
"Block": {}
},
"VisibilityConfig": {
"SampledRequestsEnabled": true,
"CloudWatchMetricsEnabled": true,
"MetricName": "user-agent"
}
}
]

You can directly "assign" to the Action element of the selected array item. |= only affects the selected item, but the output of the filter is still the entire input.
jq 'map(select(.Name=="bot-block").Action |= {Block: {}})' input.json

Super close! It's just a precedence issue.
.[] | select(.Name=="bot-block") | .Action |= . + { "Block" : {} }
should be
( .[] | select(.Name=="bot-block") | .Action ) |= . + { "Block" : {} }
because |= has lower precedence than |.
jqplay
Think of foo | bar |= baz has modifying foo | bar but returning foo. As such, the original snippet returns the modified .[] | select(.Name=="bot-block"), while the fixed version returns the modified ..

Related

jq - remove non-matching fields in "object-array-with-objects"

Given the following JSON-object:
{
"meta": {
"data1": {
"keep": { "key": "value" }
}
},
"detail": {
"data2": [
{
"keep1": "keep1value",
"keep2": "keep2value",
"nokeep1": "abc"
}
],
"data3": [
{
"keep1": "keep1value",
"keep2": "keep2value",
"nokeep2": { "abc": "def" }
}
]
},
"drop" : "this"
}
I'm trying to clean it by removing unwanted fields, like "remove", "nokeep1" and "nokeep2".
However objects in the "data2" and "data3" arrays might contain more fields than the example "nokeepX", but will always contain "keep1" and "keep2" which I want to keep.
My desired output is the following JSON:
{
"meta": { "data1": { "keep": { "key": "value" } } },
"detail": {
"data2": [
{
"keep1": "keep1value",
"keep2": "keep2value"
}
],
"data3": [
{
"keep1": "keep1value",
"keep2": "keep2value"
}
]
}
}
I've managed to remove the "drop" field with this query:
jq 'def pick($paths): . as $root | reduce ($paths[]|[.]|flatten(1)) as $path ({}; . + setpath($path; $root|getpath($path))); pick([["meta"], ["detail", "data2"], ["detail", "data3"]])'
However I've been struggling to figure out how to remove the "nokeepX" fields - is it possible to accomplish this?
If you have only a limited set of properties, it could be easier not to remove unwanted fields, but create the output from the required fields only:
{
meta,
detail: .detail | {
data2: .data2 | map({ keep1, keep2 }),
data3: .data3 | map({ keep1, keep2 })
}
}
Output:
{
"meta": {
"data1": {
"keep": {
"key": "value"
}
}
},
"detail": {
"data2": [
{
"keep1": "keep1value",
"keep2": "keep2value"
}
],
"data3": [
{
"keep1": "keep1value",
"keep2": "keep2value"
}
]
}
}
The approach can be combined with dropping certain fields:
{
meta,
detail: .detail | {
data2: .data2 | map(del(.nokeep1)),
data3: .data3 | map(del(.nokeep2))
}
}
producing the same output as above.
Just provide all the concrete paths to del:
del(
.detail.data2[0].nokeep1,
.detail.data3[0].nokeep2,
.drop
)
Demo
Or generalize by e.g. traversing all array items (not just the first) using [] without indices:
del(
.detail.data2[].nokeep1,
.detail.data3[].nokeep2,
.drop
)
Demo
Or go arbitrarily deep using .., and just provide the deepest field names:
del(.. | objects | .nokeep1, .nokeep2, .drop)
Demo
Output:
{
"meta": {
"data1": {
"keep": "true"
}
},
"detail": {
"data2": [
{
"keep1": "keep1value",
"keep2": "keep2value"
}
],
"data3": [
{
"keep1": "keep1value",
"keep2": "keep2value"
}
]
}
}
For the other way round, you could list all the leaf paths using paths(scalars), filter out those where the deepest level .[-1] does not match your criteria, and use delpaths to remove the remaining leafs:
delpaths([paths(scalars) | select(
.[-1] | IN("keep", "keep1", "keep2") | not
)])
Demo

Decrypt values with the same key at different levels from base64

My input is like below. I want to search for SearchString key (you can see that we can't use a fixed index for it) and when the key appears decrypt its value from base64 (perhaps using #base64d filter). Is this possible with JQ? If so, how?
[
{
"Name": "searchblock",
"Priority": 3,
"Statement": {
"RateBasedStatement": {
"Limit": 100,
"AggregateKeyType": "IP",
"ScopeDownStatement": {
"ByteMatchStatement": {
"SearchString": "Y2F0YWxvZ3NlYXJjaA==",
"FieldToMatch": {
"UriPath": {}
},
"TextTransformations": [
{
"Priority": 0,
"Type": "LOWERCASE"
}
],
"PositionalConstraint": "CONTAINS"
}
}
}
},
"Action": {
"Block": {}
},
"VisibilityConfig": {
"SampledRequestsEnabled": true,
"CloudWatchMetricsEnabled": true,
"MetricName": "searchblock"
}
},
{
"Name": "bot-block",
"Priority": 4,
"Statement": {
"ByteMatchStatement": {
"SearchString": "Ym90",
"FieldToMatch": {
"SingleHeader": {
"Name": "user-agent"
}
},
"TextTransformations": [
{
"Priority": 0,
"Type": "LOWERCASE"
}
],
"PositionalConstraint": "CONTAINS"
}
},
"Action": {
"Allow": {}
},
"VisibilityConfig": {
"SampledRequestsEnabled": true,
"CloudWatchMetricsEnabled": true,
"MetricName": "user-agent"
}
}
]
We use path, paths, getpath, and setpath built-ins for such operations when a fixed path is not available.
getpath(paths | select(.[-1] == "SearchString")) |= #base64d
Online demo
walk is quite intuitive for this kind of task:
walk(if type == "object" and .SearchString
then .SearchString |= #base64d else . end)
Using this approach, it's also trivial to modify the program to make it more robust, e.g. to check that .SearchString is a string:
walk(if type == "object" and (.SearchString|type) == "string"
then .SearchString |= #base64d else . end)
Note: if your jq does not include walk, you can simply copy its def from any reputable web site, or from https://github.com/stedolan/jq/blob/master/src/builtin.jq

Iterate over objects and print select values

Question: I would like to know how I can iterate over an object to print out the name of each subnet and its cidr range in the subnets object.
I know how I can print out the vnet name and cidr range:
jq '.[] | select(.vnet) | {name: .vnet.name, cidr: .vnet.address_space[0]}'
I have looked online and found that you can convert the object to an array, but I am unsure how I can look through that array to print out the values:
jq '.[] | to_entries | map_values(.value)'
Essentially the desired OUTPUT should look something like this:
{
"name": "asdf1",
"cidr": "10....."
},
{
"name": "asdf2",
"cidr": "10....."
}
//...and so forth
Sample INPUT:
{
"route_tables": {
"asdf": {
"disable_bgp_route_propagation": true,
"name": "az_afw",
"resource_group_name": "vnet-spoke",
"route_entries": {
"re1": {
"name": "rt-rfc-10-8",
"next_hop_in_ip_address": "10.0.0.0",
"next_hop_type": "VirtualAppliance",
"prefix": "10.0.0.0/8"
},
"re2": {
"name": "rt-rfc-172-12",
"next_hop_in_ip_address": "10.0.0.0",
"next_hop_type": "VirtualAppliance",
"prefix": "172.16.0.0/12"
}
}
}
},
"vnet_peering_settings": {
"peer1": {
"peer_to_source": {
"allow_forwarded_traffic": true,
"allow_gateway_transit": false,
"allow_virtual_network_access": true,
"use_remote_gateways": true
},
"source_to_peer": {
"allow_forwarded_traffic": true,
"allow_gateway_transit": true,
"allow_virtual_network_access": true,
"use_remote_gateways": false
}
}
},
"vnet_spoke_object": {
"specialsubnets": {},
"subnets": {
"objectAsdf1": {
"cidr": "10.0.0.1/24",
"enforce_private_link_endpoint_network_policies": false,
"enforce_private_link_service_network_policies": false,
"name": "asdf1",
"nsg_creation": true,
"nsg_inbound": [],
"nsg_outbound": [],
"route": null,
"service_endpoints": []
},
"objectAsdf2": {
"cidr": "10.0.0.1/24",
"enforce_private_link_endpoint_network_policies": false,
"enforce_private_link_service_network_policies": false,
"name": "asdf2",
"nsg_creation": true,
"nsg_inbound": [],
"nsg_outbound": [],
"route": "asdf",
"service_endpoints": [
"Microsoft.EventHub"
]
},
"objectAsdf3": {
"cidr": "10.0.0.1/24",
"enforce_private_link_endpoint_network_policies": false,
"enforce_private_link_service_network_policies": false,
"name": "asdf3",
"nsg_creation": true,
"nsg_inbound": [],
"nsg_outbound": [],
"route": "asdf",
"service_endpoints": []
},
"objectAsdf4": {
"cidr": "10.0.0.1/24",
"enforce_private_link_endpoint_network_policies": false,
"enforce_private_link_service_network_policies": false,
"name": "asdf4",
"nsg_creation": true,
"nsg_inbound": [],
"nsg_outbound": [],
"route": "asdf",
"service_endpoints": []
}
},
"vnet": {
"address_space": [
"10.0.0.0/16"
],
"ddos_id": "placeholder",
"dns": [
"10.0.0.1",
"10.0.0.1"
],
"enable_ddos_std": false,
"name": "asdf"
}
}
}
Start with the root path and traverse down and use to_entries to get past the varying key names and use the .value field only
jq '.vnet_spoke_object.subnets | to_entries[].value | { name, cidr }'
or collect it as an array of objects
jq '.vnet_spoke_object.subnets | to_entries | map(.value | { name, cidr })'
jqplay - Demo

Reconstructing JSON with jq

I have a JSON like this (sample.json):
{
"sheet1": [
{
"hostname": "sv001",
"role": "web",
"ip1": "172.17.0.3"
},
{
"hostname": "sv002",
"role": "web",
"ip1": "172.17.0.4"
},
{
"hostname": "sv003",
"role": "db",
"ip1": "172.17.0.5",
"ip2": "172.18.0.5"
}
],
"sheet2": [
{
"hostname": "sv004",
"role": "web",
"ip1": "172.17.0.6"
},
{
"hostname": "sv005",
"role": "db",
"ip1": "172.17.0.7"
},
{
"hostname": "vsv006",
"role": "db",
"ip1": "172.17.0.8"
}
],
"sheet3": []
}
I want to extract data like this:
sheet1
jq '(something command)' sample.json
{
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
},
"db": {
"hosts": [
"172.17.0.5"
]
}
}
Is it possible to perform the reconstruction with jq map?
(I will reuse the result for ansible inventory.)
Here's a short, straight-forward and efficient solution -- efficient in part because it avoids group_by by courtesy of the following generic helper function:
def add_by(f;g): reduce .[] as $x ({}; .[$x|f] += [$x|g]);
.sheet1
| add_by(.role; .ip1)
| map_values( {hosts: .} )
Output
This produces the required output:
{
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
},
"db": {
"hosts": [
"172.17.0.5"
]
}
}
If the goal is to regroup the ips by their roles within each sheet you could do this:
map_values(
reduce group_by(.role)[] as $g ({};
.[$g[0].role].hosts = [$g[] | del(.hostname, .role)[]]
)
)
Which produces something like this:
{
"sheet1": {
"db": {
"hosts": [
"172.17.0.5",
"172.18.0.5"
]
},
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
}
},
"sheet2": {
"db": {
"hosts": [
"172.17.0.7",
"172.17.0.8"
]
},
"web": {
"hosts": [
"172.17.0.6"
]
}
},
"sheet3": {}
}
https://jqplay.org/s/3VpRc5l4_m
If you want to flatten all to a single object keeping only unique ips, you can keep everything mostly the same, you'll just need to flatten the inputs prior to grouping and remove the map_values/1 call.
$ jq -n '
reduce ([inputs[][]] | group_by(.role)[]) as $g ({};
.[$g[0].role].hosts = ([$g[] | del(.hostname, .role)[]] | unique)
)
'
{
"db": {
"hosts": [
"172.17.0.5",
"172.17.0.7",
"172.17.0.8",
"172.18.0.5"
]
},
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4",
"172.17.0.6"
]
}
}
https://jqplay.org/s/ZGj1wC8hU3

JQ | Updating array element selected by `select`

In a JSON array, I want to select an array element on basis of a node's value, then update a different node in the same array element. E.g. in the JSON below:
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorRelease": "1.7",
"orchestratorVersion": "1.7.10",
"kubernetesConfig": {
"kubernetesImageBase": "gcrio.azureedge.net/google_containers/",
"clusterSubnet": "10.105.208.0/20",
"networkPolicy": "calico",
"nonMasqueradeCidr": "10.0.0.0/8",
"maxPods": 110,
"dockerBridgeSubnet": "172.17.0.1/16"
"addons": [
{
"name": "tiller",
"enabled": true
},
{
"name": "aci-connector",
"enabled": true
},
{
"name": "kubernetes-dashboard",
"enabled": true
},
{
"name": "rescheduler",
"enabled": true
}
]
}
}
}
}
I want to disable all addons which are not "rescheduler", i.e. set .enabled = false for elements of array .properties.orchestratorProfile.kubernetesConfig.addons[] where .name != "rescheduler". Closest I could work out was
jq -r '.properties.orchestratorProfile.kubernetesConfig.addons[] |
select (.name != "rescheduler" ) | .enabled = false'
but this, or any other ways I tried, I always lose the data outside of the array.
The expected outcome is:
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorRelease": "1.7",
"orchestratorVersion": "1.7.10",
"kubernetesConfig": {
"kubernetesImageBase": "gcrio.azureedge.net/google_containers/",
"clusterSubnet": "10.105.208.0/20",
"networkPolicy": "calico",
"nonMasqueradeCidr": "10.0.0.0/8",
"maxPods": 110,
"dockerBridgeSubnet": "172.17.0.1/16"
"addons": [
{
"name": "tiller",
"enabled": false
},
{
"name": "aci-connector",
"enabled": false
},
{
"name": "kubernetes-dashboard",
"enabled": false
},
{
"name": "rescheduler",
"enabled": true
}
]
}
}
}
}
How do I go about doing this? Any idea or help or guidance is appreciated in advance.
Your jq query is spot-on except essentially for a missing pair of parentheses:
(.properties.orchestratorProfile.kubernetesConfig.addons[]
| select (.name != "rescheduler" ).enabled) = false
That is, on the LHS of the assignment, you need to specify the paths of the values that need to be updated.
jq solution:
jq '.properties.orchestratorProfile.kubernetesConfig.addons =
[.[] | if .name != "rescheduler" then .enabled = false else . end]' file