JQ How to select multiple values when key matches - json

Looking somebody who can help me with the following jq manipulation. I have the following json:
{
"description": "James-only access",
"groupName": "Route-SGP",
"ipPermissions": [
{
"ipProtocol": "tcp",
"toPort": 22,
"ipRanges": [
"10.32.0.1/32",
"10.32.0.2/32"
]
}
],
"ownerId": "2222",
"groupId": "sg-2222",
"vpcId": "vpc-2222"
}
{
"groupName": "sg1",
"ipPermissions": [
{
"ipProtocol": "tcp",
"toPort": 4439,
"ipRanges": [
"10.32.0.2/32"
]
},
{
"ipProtocol": "tcp",
"toPort": 3389,
"ipRanges": [
"10.32.0.1/32",
"10.32.0.2/32"
]
},
{
"ipProtocol": "icmp",
"toPort": -1,
"ipRanges": [
"10.32.0.0/30"
]
}
],
"ownerId": "1111",
"groupId": "sg-1111",
"vpcId": "vpc-1111"
}
I need get the following keys when ipRanges contains "10.32.0.2/32", like that:
ownerId, groupId, groupName, ipProtocol, toPort
"2222","sg-2222","Route-SGP","tcp","22"
"1111","sg-1111","sg1","tcp","4439"
"1111","sg-1111","sg1","tcp","3389"
I tried do something like below but it wont work.
jq 'select(.ipPermissions[]?.ipRanges[] | contains("10.32.0.2/32"))' | jq 'del(.[] | first(select(recurse | objects | has("ipRanges") and (.ipRanges | index("10.32.0.2/32" | not)))))'

Iterate using a variable, and use select to filter
jq -r --arg ip "10.32.0.2/32" '
.ipPermissions[] as $p | select(IN($p.ipRanges[]; $ip))
| [.ownerId, .groupId, .groupName, $p.ipProtocol, $p.toPort]
| #csv
'
"2222","sg-2222","Route-SGP","tcp",22
"1111","sg-1111","sg1","tcp",4439
"1111","sg-1111","sg1","tcp",3389
Demo

I think i figured out:
jq 'select(.ipPermissions[]?.ipRanges[] | index("10.32.0.2/32"))' | jq 'del(.ipPermissionsEgress) | . as $i| "\(.ownerId),\(.groupId),\($i.ipPermissions[] | select(.ipRanges | index("10.32.0.2/32")) | "\(.toPort),\(.ipProtocol)")"'

Related

parsing jq returns null

I have a json output
{
"7": [
{
"devices": [
"/dev/sde"
],
"name": "osd-block-dcc9b386-529c-451e-9d84-8ccc4091102b",
"tags": {
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/nvme0n1p5",
"ceph.wal_device": "/dev/nvme0n1p6",
},
"type": "block",
"vg_name": "ceph-c4de9e90-853e-4569-b04f-8677ef9a8c7a"
},
{
"path": "/dev/nvme0n1p5",
"tags": {
"PARTUUID": "69712eb4-be52-4618-ba46-e317d6d3d76e"
},
"type": "db"
}
],
"41": [
{
"devices": [
"/dev/nvme1n1p13"
],
"name": "osd-block-97bce07f-ae98-4fdb-83a9-9fa2f35cee60",
"tags": {
"ceph.crush_device_class": "None",
},
"type": "block",
"vg_name": "ceph-c1d48671-2a33-4615-95e3-cc1b18783f0c"
}
],
"9": [
{
"devices": [
"/dev/sdf"
],
"name": "osd-block-35323eb8-17c1-460d-8cc5-565f549e6991",
"tags": {
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/nvme0n1p7",
"ceph.wal_device": "/dev/nvme0n1p8",
},
"type": "block",
"vg_name": "ceph-9488e8b8-ec18-4860-93d3-6a1ad91c698c"
},
{
"path": "/dev/nvme0n1p7",
"tags": {
"PARTUUID": "ef0e9588-2a20-4c2c-8b62-d73945e01322"
},
"type": "db"
}
]
}
Required output:
osd.7 /dev/sde /dev/nvme0n1p5 /dev/nvme0n1p6
osd.41 /dev/nvme1n1p13 n/a n/a
osd.9 /dev/sdf /dev/nvme0n1p7 /dev/nvme0n1p7
Problems:
When I try parsing using jq .[][].devices, I get null values:
$ cat json | jq .[][].devices
[
"/dev/sde"
]
null
[
"/dev/nvme1n1p13"
]
null
[
"/dev/sdf"
]
null
I can solve it via jq .[][].devices[]?.
However, this trick doesn't help me when I do want to see where there's no value (to print n/a instead):
$ cat json | jq '.[][].tags | ."ceph.db_device"'
"/dev/nvme0n1p5"
null
"/dev/nvme0n1p3"
null
null
"/dev/nvme0n1p7"
null
And finally, I try to create a table:
$ cat json | jq -r '["osd."+keys[]], [.[][].devices[]?], [.[][].tags."ceph.db_device" // ""] | #csv' | column -t -s,
"osd.7" "osd.41" "osd.9"
"/dev/sde" "/dev/nvme0n1p13" "/dev/sdf"
"/dev/nvme0n1p5" "/dev/nvme0n1p7"
So the obvious problem is that the 3rd row doesn't match the correct values.
And the final problem is how do I transpose it from columns to rows, as detailed in the required output?
Would this do what you want?
jq --raw-output '
to_entries[] | [
"osd." + .key,
( .value[0]
| .devices[],
( .tags
| ."ceph.db_device" // "n/a",
."ceph.wal_device" // "n/a"
)
)
]
| #tsv
'
osd.7 /dev/sde /dev/nvme0n1p5 /dev/nvme0n1p6
osd.41 /dev/nvme1n1p13 n/a n/a
osd.9 /dev/sdf /dev/nvme0n1p7 /dev/nvme0n1p8
Demo

How i can get needest line " DBInstanceIdentifier" from Json output with jq or other tools

this is Json: I try to use
jq -r '.[] | .DBSnapshotIdentifier'
But it doesn't work for me
{
"DBSnapshots": [
{
"DBSnapshotIdentifier": "auto-lims-final-snapshot",
"DBInstanceIdentifier": "auto-lims",
"SnapshotCreateTime": "2018-08-15T09:59:23.332000+00:00",
"Engine": "postgres",
"AllocatedStorage": 50,
"Status": "available",
"Port": 5432,
"AvailabilityZone": "us-east-2a",
"VpcId": "vpc-e799fc8f",
"InstanceCreateTime": "2018-04-09T08:28:03.565000+00:00",
"MasterUsername": "postgres",
"EngineVersion": "9.6.6",
"LicenseModel": "postgresql-license",
"SnapshotType": "manual",
"OptionGroupName": "default:postgres-9-6",
"PercentProgress": 100,
"StorageType": "gp2",
"Encrypted": false,
"DBSnapshotArn": "arn:aws:rds:us-east-2:833682533595:snapshot:auto-lims-final-snapshot",
"IAMDatabaseAuthenticationEnabled": false,
"ProcessorFeatures": [],
"DbiResourceId": "db-ZL5T2TA2PJVG6CVOJRO7HUOAXQ",
"TagList": [
{
"Key": "Environment",
"Value": "auto"
},
{
"Key": "Application",
"Value": "LIMS"
}
],
"SnapshotTarget": "region"
},
{
"DBSnapshotIdentifier": "automation-lims-before-postgres-12-5",
"DBInstanceIdentifier": "automation-lims",
"Engine": "postgres",
"AllocatedStorage": 500,
"Status": "available",
"Port": 5432,
"AvailabilityZone": "us-east-2b",
"VpcId": "vpc-09fa88d2884ee2083",
"InstanceCreateTime": "2019-12-26T11:19:41.947000+00:00",
"MasterUsername": "lims",
"EngineVersion": "9.6.20",
"LicenseModel": "postgresql-license",
"SnapshotType": "manual",
"OptionGroupName": "default:postgres-9-6",
"PercentProgress": 100
}
]
}
How I can get only nested lines.
You first need to select the DBSnapshots object, try:
.DBSnapshots[] | .DBSnapshotIdentifier
This will output:
"auto-lims-final-snapshot"
"automation-lims-before-postgres-12-5"
As you can try in this online demo
To include the .TagList.Key we can add the following:
.DBSnapshots[] | .DBSnapshotIdentifier, .TagList[]?.Key
This will now output
"auto-lims-final-snapshot"
"Environment"
"Application"
"automation-lims-before-postgres-12-5"
As you can try in this online demo
If you'd like the DBSnapshotIdentifier to be on the same line as the TagList we can use string interpolation like so:
.DBSnapshots[] | "\(.DBSnapshotIdentifier) - \(.TagList[]?.Key)"
Thiis will output:
"auto-lims-final-snapshot - Environment"
"auto-lims-final-snapshot - Application"
As this demo demostrates
The .[]? from the above command mean:
Like .[], but no errors will be output if . is not an array or object.
More information in JQ's documentation

Reconstructing JSON with jq

I have a JSON like this (sample.json):
{
"sheet1": [
{
"hostname": "sv001",
"role": "web",
"ip1": "172.17.0.3"
},
{
"hostname": "sv002",
"role": "web",
"ip1": "172.17.0.4"
},
{
"hostname": "sv003",
"role": "db",
"ip1": "172.17.0.5",
"ip2": "172.18.0.5"
}
],
"sheet2": [
{
"hostname": "sv004",
"role": "web",
"ip1": "172.17.0.6"
},
{
"hostname": "sv005",
"role": "db",
"ip1": "172.17.0.7"
},
{
"hostname": "vsv006",
"role": "db",
"ip1": "172.17.0.8"
}
],
"sheet3": []
}
I want to extract data like this:
sheet1
jq '(something command)' sample.json
{
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
},
"db": {
"hosts": [
"172.17.0.5"
]
}
}
Is it possible to perform the reconstruction with jq map?
(I will reuse the result for ansible inventory.)
Here's a short, straight-forward and efficient solution -- efficient in part because it avoids group_by by courtesy of the following generic helper function:
def add_by(f;g): reduce .[] as $x ({}; .[$x|f] += [$x|g]);
.sheet1
| add_by(.role; .ip1)
| map_values( {hosts: .} )
Output
This produces the required output:
{
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
},
"db": {
"hosts": [
"172.17.0.5"
]
}
}
If the goal is to regroup the ips by their roles within each sheet you could do this:
map_values(
reduce group_by(.role)[] as $g ({};
.[$g[0].role].hosts = [$g[] | del(.hostname, .role)[]]
)
)
Which produces something like this:
{
"sheet1": {
"db": {
"hosts": [
"172.17.0.5",
"172.18.0.5"
]
},
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
}
},
"sheet2": {
"db": {
"hosts": [
"172.17.0.7",
"172.17.0.8"
]
},
"web": {
"hosts": [
"172.17.0.6"
]
}
},
"sheet3": {}
}
https://jqplay.org/s/3VpRc5l4_m
If you want to flatten all to a single object keeping only unique ips, you can keep everything mostly the same, you'll just need to flatten the inputs prior to grouping and remove the map_values/1 call.
$ jq -n '
reduce ([inputs[][]] | group_by(.role)[]) as $g ({};
.[$g[0].role].hosts = ([$g[] | del(.hostname, .role)[]] | unique)
)
'
{
"db": {
"hosts": [
"172.17.0.5",
"172.17.0.7",
"172.17.0.8",
"172.18.0.5"
]
},
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4",
"172.17.0.6"
]
}
}
https://jqplay.org/s/ZGj1wC8hU3

Convert/Export JSON to CSV

JSON file:
"UserDetailList": [
{
"UserName": "citrix-xendesktop-ec2-provisioning",
"GroupList": [],
"CreateDate": "2017-11-07T14:20:14Z",
"UserId": "1234556",
"Path": "/",
"AttachedManagedPolicies": [
{
"PolicyName": "AmazonEC2FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
},
{
"PolicyName": "AmazonS3FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
],
"Arn": "arn:aws:iam::1234567890:user/citrix-xendesktop-ec2-provisioning"
},
{
"UserName": "rundeck-read-only-iam-permissions",
"GroupList": [],
"CreateDate": "2018-03-09T11:13:38Z",
"UserId": "AIDAJQOQGKISLCWDXG6EQ",
"Path": "/",
"AttachedManagedPolicies": [
{
"PolicyName": "IAMReadOnlyAccess",
"PolicyArn": "arn:aws:iam::aws:policy/IAMReadOnlyAccess"
}
],
"Arn": "arn:aws:iam::279052847476:user/rundeck-read-only-iam-permissions"
}
]
with
jq -r '.UserDetailList[] | [.UserName] | #csv' output.json > fileout2.csv
I can get
citrix-xendesktop-ec2-provisioning"
"rundeck-read-only-iam-permissions"
How to get IAM policies for these 2 users, i need to extract AmazonEC2FullAccess and AmazonS3FullAccess under AttachedManagedPolicies ?
so output can be
citrix-xendesktop-ec2-provisioning",AmazonEC2FullAccess
citrix-xendesktop-ec2-provisioning",AmazonS3FullAccess
rundeck-read-only-iam-permissions,IAMReadOnlyAccess
The trick is to extract .UserName as a variable before iterating over the inner array:
.UserDetailList[]
| .UserName as $u
| .AttachedManagedPolicies[]
| [$u, .PolicyName]
| #csv
Of course this assumes valid JSON input.

jq: output values of ids instead of numbers

Here's my input json:
{
"channels": [
{ "id": 1, "name": "Pop"},
{ "id": 2, "name": "Rock"}
],
"links": [
{ "id": 2, "streams": [ {"url": "http://example.com/rock"} ] },
{ "id": 1, "streams": [ {"url": "http://example.com/pop"} ] }
]
}
This is what I want as an output:
"http://example.com/pop"
"Pop"
"http://example.com/rock"
"Rock"
So I need jq to replace .channels[].id with .links[].streams[0].url based on .links[].id
I don't know if it's right, but this is how I managed to output the urls:
(.channels[].id | tostring) as $ids | [.links[]] | map({(.id | tostring): .streams[0].url}) | add as $urls | $urls[$ids]
"http://example.com/pop"
"http://example.com/rock"
The question is, how do I add .channels[].name to it?
You sometimes have to be careful what you ask for, but this will produce the result you said you want:
.channels[] as $channel
| $channel.name,
(.links[] | select(.id == $channel.id) | .streams[0].url)
Output for the given input:
"Pop"
"http://example.com/pop"
"Rock"
"http://example.com/rock"
Here is a solution which uses reduce and setpath to make a $urls lookup table from .links and then scans .channels generating corresponding urls and names.
(
reduce .links[] as $l (
{};
setpath([ $l.id|tostring ]; [$l.streams[].url])
)
) as $urls
| .channels[]
| $urls[ .id|tostring ][], .name
If multiple urls are present in the "streams" attribute this will
print them all before printing the name. e.g. if the input is
{
"channels": [
{ "id": 1, "name": "Pop"},
{ "id": 2, "name": "Rock"}
],
"links": [
{ "id": 2, "streams": [ {"url": "http://example.com/rock"},
{"url": "http://example.com/hardrock"} ] },
{ "id": 1, "streams": [ {"url": "http://example.com/pop"} ] }
]
}
the output will be
"http://example.com/pop"
"Pop"
"http://example.com/rock"
"http://example.com/hardrock"
"Rock"