Parsing aws ec2 describe-instances to get all private ip addresses - json

Question

Two of my EC2 instances have 3 IPs each. I managed to successfully grab a list of JSON objects:
aws ec2 describe-instances | jq '.Reservations[] | .Instances[] | (.Tags | { "iname": ( map ( select(.Value | contains("my-vm")))[] | .Value ) } ) + ( { "ip": ( .NetworkInterfaces[].PrivateIpAddress) } )' | jq -s .
Giving me the following result:
[
{
"iname": "my-vm-b",
"ip": "10.11.2.145"
},
{
"iname": "my-vm-b",
"ip": "10.11.1.146"
},
{
"iname": "my-vm-b",
"ip": "10.11.10.144"
},
{
"iname": "my-vm-a",
"ip": "10.11.1.9"
},
{
"iname": "my-vm-a",
"ip": "10.11.10.125"
},
{
"iname": "my-vm-a",
"ip": "10.11.2.85"
}
]
and then I added to the command the following:
... | jq ' group_by(.iname)[] | {(.[0].iname): [.[] | .ip]}' | jq -s .
To finally get the list of objects the way I wanted:
[
{
"my-vm-a": [
"10.11.1.9",
"10.11.10.125",
"10.11.2.85"
]
},
{
"my-vm-b": [
"10.11.2.145",
"10.11.1.146",
"10.11.10.144"
]
}
]
Notice I had to call jq like 4 times. I know I must be doing something wrong so I was wondering if I could do it with a single jq call.
Thanks!

Answer 1

You can easily eliminate the calls to jq -s by wrapping expressions as appropriately in square brackets, or maybe better, using map.
For example, your last pair of jq calls can be simplified to:
jq 'group_by(.iname) | map({(.[0].iname): [.[] | .ip]})'
The following should allow you to reduce the four calls to one:
[.Reservations[]
| .Instances[]
| (.Tags | { "iname": ( map ( select(.Value | contains("my-vm")))[] | .Value ) } )
+ ( { "ip": ( .NetworkInterfaces[].PrivateIpAddress) } ) ]
| group_by(.iname) | map({(.[0].iname): [.[] | .ip]})
However, I would advise against using contains here, unless you fully understand the complications.

Answer 2

Before you go into trying to simplify your jq calls, I think it would be more beneficial to first look at the source data and how it relates to the result you want.
Ignoring a lot of the other details in the data, I think we can agree that your data looks sorta like this:
{
"Reservations": [
{
"Instances": [
{
"NetworkInterfaces": [
{ "PrivateIpAddress": "10.11.2.145" },
{ "PrivateIpAddress": "10.11.1.146" },
{ "PrivateIpAddress": "10.11.10.144" }
],
"Tags": [
{ "Key": "Name", "Value": "my-vm-b" }
]
},
{
"NetworkInterfaces": [
{ "PrivateIpAddress": "10.11.1.9" },
{ "PrivateIpAddress": "10.11.10.125" },
{ "PrivateIpAddress": "10.11.2.85" }
],
"Tags": [
{ "Key": "Name", "Value": "my-vm-a" }
]
}
]
}
]
}
With something that looks like this, your jq query can simply be:
[.Reservations[].Instances[] |
{
(.Tags|from_entries.Name): [.NetworkInterfaces[].PrivateIpAddress]
}
]
No intermediate results needed. Just a few things of note here.
The tags are already an array of key/value pairs, you can easily read values from here converting them to an object first using from_entries
You are selecting instances based on an existence of a tag value containing "my-vm". I'm not sure you even need to do this, I don't know what your data looks like but they are likely in a fixed name so you should just use that name.