This question already has answers here:
Get or default function in JQ?
(2 answers)
Closed 2 years ago.
I have the following JSON:
{
"SecurityGroups": [
{
"Description": "launch-wizard-6 created 2018-10-25T13:59:20.092+03:00",
"GroupName": "launch-wizard-6",
"IpPermissions": [
{
"IpProtocol": "-1",
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"UserIdGroupPairs": []
},
{
"FromPort": 22,
"IpProtocol": "tcp",
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"ToPort": 22,
"UserIdGroupPairs": []
}
],
"OwnerId": "XXXXXXXXXXXX",
"GroupId": "sg-054368f50a6b4fea4",
"IpPermissionsEgress": [
{
"IpProtocol": "-1",
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [
{
"CidrIpv6": "::/0"
}
],
"PrefixListIds": [],
"UserIdGroupPairs": []
}
],
"VpcId": "vpc-6ea82a08"
}
]
}
When I run the following jq command:
aws ec2 describe-security-groups --group-id ${group_id} --profile ${profile} --region ${region} --output json | jq -r '.SecurityGroups[].IpPermissions[] | [ (.FromPort|tostring), .IpProtocol, .IpRanges[].CidrIp // .UserIdGroupPairs[].GroupId // "" ] | #tsv'
I get the following output:
null -1 0.0.0.0/0
22 tcp 0.0.0.0/0
I'd like to remove the null from the output, how can it be done?
Add a fallback for the .FromPort key:
.FromPort // ""
So the command becomes;
.SecurityGroups[].IpPermissions[] | [ (.FromPort // "" ?), .IpProtocol, .IpRanges[].CidrIp ] | #tsv
Which yields to
-1 0.0.0.0/0
22 tcp 0.0.0.0/0
Try it online!
Related
As part of a shell script I'm writing, I'm querying AWS (cli) to pull information regarding available security group names and ids, like so:
aws ec2 describe-security-groups | jq -r '.SecurityGroups[]'
{
"IpPermissionsEgress": [
{
"IpProtocol": "-1",
"PrefixListIds": [],
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"UserIdGroupPairs": [],
"Ipv6Ranges": []
}
],
"Description": "default VPC security group",
"IpPermissions": [
{
"PrefixListIds": [],
"FromPort": 80,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 80,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": [
{
"CidrIpv6": "::/0"
}
]
},
{
"PrefixListIds": [],
"FromPort": 22,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 22,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": -1,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": -1,
"IpProtocol": "icmp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
}
],
"GroupName": "default",
"VpcId": "vpc-b3c29bcb",
"OwnerId": "506490286752",
"GroupId": "sg-83db2ef7"
}
And using jq, I'm trying to return a list of maps which displays the info like so:
GroupName , GroupId
This is what I've tried:
aws ec2 describe-security-groups | jq -r '.SecurityGroups[] | to_entries[] | [ .GroupName.value , .GroupId.value]'
Using the above method returns:
[
null,
null
]
[
null,
null
]
In the example, the delimiter is "," but I'd like the displayed output to be like so (example):
"default - sg-abd837s"
How can it be done by using jq?
Using string interpolation:
.SecurityGroups[] | "\(.GroupName) - \(.GroupId)"
There should be no need to use to_entries:
.SecurityGroups[]
| [ .GroupName, .GroupId ]
| join(" - ")
produces:
"default - sg-abd837s"
I have some json that I need to parse using jq. I am trying to extract the
SecurityGroups>GroupId if any of the SecurityGroups>IpPermissions>IpRanges>CidrIp matches a certain IP.
For instance, searching for 11.11.11.11 should return sg-3jf32kj3j. There is a possibility that multiple SecurityGroups will contain that IP. I need to return every GroupId that does.
Is this even possible with jq alone or will it require bash as well?
I'm finding jq syntax confusing compared to just doing this with something like Python.
{
"SecurityGroups": [{
"OwnerId": "111111111",
"Description": "default VPC security group",
"GroupId": "sg-1a1a1a1a1",
"VpcId": "vpc-1a1a1a1a1",
"IpPermissionsEgress": [{
"IpProtocol": "-1",
"PrefixListIds": [],
"Ipv6Ranges": [],
"UserIdGroupPairs": [],
"IpRanges": [{
"CidrIp": "0.0.0.0/0"
}]
}],
"GroupName": "default",
"IpPermissions": [{
"IpProtocol": "-1",
"PrefixListIds": [],
"Ipv6Ranges": [],
"UserIdGroupPairs": [{
"GroupId": "sg-5df45d5d5",
"UserId": "234234234"
}],
"IpRanges": []
}]
},
{
"OwnerId": "22222222222",
"Description": "EC2 Security Group",
"Tags": [{
"Key": "aws:cloudformation:logical-id",
"Value": "EC2SecurityGroup"
},
{
"Key": "aws:cloudformation:stack-id",
"Value": "arn:aws:cloudformation:us-west-2:111111111:stack/blah-prod-vpc/asdfsdf-j3j3-22j1-39fj-sadfsadf"
},
{
"Key": "Name",
"Value": "blah-production-EC2"
},
{
"Key": "Owner",
"Value": "blah#blah.com"
},
{
"Key": "aws:cloudformation:stack-name",
"Value": "prod-vpc"
},
{
"Key": "Created",
"Value": "2018-05-21T09:40:55-07:00"
}
],
"GroupId": "sg-3jf32kj3j",
"VpcId": "vpc-3kj3f2kj3",
"IpPermissionsEgress": [{
"IpProtocol": "-1",
"PrefixListIds": [],
"Ipv6Ranges": [],
"UserIdGroupPairs": [],
"IpRanges": [{
"CidrIp": "0.0.0.0/0"
}]
}],
"GroupName": "blah-prod-vpc-EC2SecurityGroup-SJHS78F78SSH",
"IpPermissions": [{
"IpProtocol": "tcp",
"ToPort": 80,
"UserIdGroupPairs": [],
"IpRanges": [{
"CidrIp": "0.0.0.0/0"
}],
"FromPort": 80,
"PrefixListIds": [],
"Ipv6Ranges": []
},
{
"IpProtocol": "icmp",
"ToPort": 0,
"UserIdGroupPairs": [],
"IpRanges": [{
"CidrIp": "11.11.11.11/32"
}],
"FromPort": 0,
"PrefixListIds": [],
"Ipv6Ranges": []
},
{
"IpProtocol": "-1",
"PrefixListIds": [],
"Ipv6Ranges": [],
"UserIdGroupPairs": [],
"IpRanges": [{
"CidrIp": "22.22.22.22/16"
},
{
"CidrIp": "33.33.33.33/32"
}
]
},
{
"IpProtocol": "tcp",
"ToPort": 22,
"UserIdGroupPairs": [],
"IpRanges": [{
"CidrIp": "0.0.0.0/0"
}],
"FromPort": 22,
"PrefixListIds": [],
"Ipv6Ranges": [{
"CidrIpv6": "::/0"
}]
},
{
"IpProtocol": "tcp",
"ToPort": 443,
"UserIdGroupPairs": [],
"IpRanges": [{
"CidrIp": "0.0.0.0/0"
}],
"FromPort": 443,
"PrefixListIds": [],
"Ipv6Ranges": []
},
{
"IpProtocol": "icmp",
"ToPort": -1,
"UserIdGroupPairs": [],
"IpRanges": [{
"CidrIp": "44.44.44.44/32"
},
{
"CidrIp": "55.55.55.55/29"
}
],
"FromPort": -1,
"PrefixListIds": [],
"Ipv6Ranges": []
}
]
}
]}
The following filter meets the stated requirements:
.SecurityGroups[]
| select( any(.IpPermissions[].IpRanges[];
.CidrIp | startswith("11.11.11.11/") ) )
| .GroupId
In particular, with your input it yields the required value:
"sg-3jf32kj3j"
You might want to consider more sophisticated pattern-matching, perhaps using:
test("^11.11.11.11($|/)")
If you want the quotation marks to be dropped, consider using the -r command-line option of jq.
Using ..
Here's another solution, but please note that it has completely different semantics:
..
| select( .. | test("^11.11.11.11($|/)")? )
| .GroupId? // empty
may I offer you an alternative solution to your ask? this is how it looks like with the unix utility jtc I have developed recently:
bash $ cat file.json | jtc -w'<^11.11.11.11\/>R: [-5] [GroupId]'
"sg-3jf32kj3j"
bash $
it will also display all groups where this ip address shows up. If you like it, you can find the utility on github.com or google it up with jtc and json keywords
Basically I'm trying to iterate through my AWS security groups to find any CIDR's using 0.0.0.0/0.
Here is my example JSON file:
{
"SecurityGroups": [
{
"IpPermissionsEgress": [],
"Description": "AWS OpsWorks load balancer - do not change or delete",
"IpPermissions": [
{
"PrefixListIds": [],
"FromPort": 22,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 22,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": 80,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 80,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": 443,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 443,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
}
],
"GroupName": "AWS-OpsWorks-LB-Server",
"OwnerId": "056146032236",
"GroupId": "sg-7dd13739"
},
{
"IpPermissionsEgress": [
{
"IpProtocol": "-1",
"PrefixListIds": [],
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"UserIdGroupPairs": [],
"Ipv6Ranges": []
}
],
"Description": "SG for bastion hosts",
"Tags": [
{
"Value": "bastion-host-sg",
"Key": "Name"
}
],
"IpPermissions": [
{
"PrefixListIds": [],
"FromPort": 80,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 80,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": 1991,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 1991,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": 8080,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 8080,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": 1194,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 1194,
"IpProtocol": "udp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": 22,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 22,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": [
{
"CidrIpv6": "::/0"
}
]
},
{
"PrefixListIds": [],
"FromPort": 30,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": -1,
"IpProtocol": "icmp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": 1194,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 1194,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": 53,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 53,
"IpProtocol": "udp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": 53,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 53,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": 443,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 443,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": 8,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": -1,
"IpProtocol": "icmp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
}
],
"GroupName": "bastion-host-sg",
"VpcId": "vpc-effd0e8a",
"OwnerId": "056146032236",
"GroupId": "sg-0f60196a"
}
]
}
Here is the command I'm trying to run, but get an error:
$ cat sg-small.json | jq '.SecurityGroups[].IpPermissions[].IpRanges[] | map(select(any(.CidrIp == "0.0.0.0/0")))'
jq: error (at <stdin>:227): Cannot iterate over string ("0.0.0.0/0")
Any thoughts as to why this is happening. That method seems to work for other things that aren't IP addresses.
Your expression:
.SecurityGroups[].IpPermissions[].IpRanges[]
is just a stream of CidrIp objects, so that's not what you want.
The following will select the SecurityGroups that meet the criterion:
.SecurityGroups[]
| select(any( .IpPermissions[].IpRanges[]; .CidrIp == "0.0.0.0/0"))
Whether this is precisely what you want is unclear since you haven't specified that. Please see http://stackoverflow.com/help/mcve
The reason you're seeing the error
Cannot iterate over string ("0.0.0.0/0")
is because with your data the first portion of your filter
.SecurityGroups[].IpPermissions[].IpRanges[]
generates a sequence of objects
{
"CidrIp": "0.0.0.0/0"
}
...
The map portion of your filter iterates over the values in each of these objects, passing
"0.0.0.0/0"
to the expression select(any(.CidrIp == "0.0.0.0/0")) where any tries to iterate over all the values within the string and fails with the error you observe.
If you only want to see a sequence of {"CidrIp":...} objects as above you can eliminate the map and any:
.SecurityGroups[].IpPermissions[].IpRanges[]
| select(.CidrIp == "0.0.0.0/0")
If you would rather collect those objects into an array you could remove the any and move some of the iteration into the map e.g.
.SecurityGroups
| map(.IpPermissions[].IpRanges[] | select(.CidrIp == "0.0.0.0/0"))
producing
[
{
"CidrIp": "0.0.0.0/0"
},
....
The explict construction of the result array is easier to see if you replace map with its definition. Since map(f) is defined as [ .[] | f ] the above filter is the same as this one:
.SecurityGroups
| [ .[] | .IpPermissions[].IpRanges[] | select(.CidrIp == "0.0.0.0/0") ]
I am trying to parse convert Json Task definition with new image.
I want to change the "image": "docker.org/alpha/alpha-app-newgen:12.2.3" value in Json to "image": "docker.org/alpha/alpha-app-newgen:12.5.0" or any other version dynamically.
below is my Json task def:
{
"taskDefinition": {
"family": "ing-stack",
"volumes": [
{
"host": {
"sourcePath": "/tmp/nginx/elb.conf"
},
"name": "volume-0"
}
],
"containerDefinitions": [
{
"dnsSearchDomains": [],
"environment": [
{
"name": "API_SECRET",
"value": "ING-SECRET"
},
{
"name": "API_KEY",
"value": "AVERA-CADA-VERA-KEY"
}
],
"readonlyRootFilesystem": false,
"name": "ing-stg",
"links": [],
"mountPoints": [],
"image": "docker.org/alpha/alpha-app-newgen:12.2.3",
"privileged": false,
"essential": true,
"portMappings": [
{
"protocol": "tcp",
"containerPort": 19000,
"hostPort": 19000
}
],
"dockerLabels": {}
},
{
"dnsSearchDomains": [],
"environment": [
{
"name": "NG_PROXY",
"value": "ing"
}
],
"readonlyRootFilesystem": false,
"name": "web",
"links": [
"identity-ng"
],
"mountPoints": [
{
"sourceVolume": "volume-0",
"readOnly": false,
"containerPath": "/etc/nginx/conf.d/default.conf"
}
],
"image": "docker.org/alpha/alpha-ui:6.4.7",
"portMappings": [
{
"protocol": "tcp",
"containerPort": 443,
"hostPort": 443
},
{
"protocol": "tcp",
"containerPort": 80,
"hostPort": 80
}
],
"memory": 512,
"command": [
"sh",
"prep-run-nginx.sh"
],
"dockerLabels": {}
}
],
"revision": 136
}
}
I need to get the same structure back with new value for the image.
I tried the following
jq '. | select(.containerDefinitions[].image | contains("'$new_img_no_ver'") ) | .image |= "my new image"', but its adding to the end of the JSON.
Can anybody tell me how to achieve this.
Here are two potential solutions. Other variants are of course possible.
walk/1
If you don't want to be bothered with the details of exactly where the relevant "image" tag is located, consider using walk/1:
Invocation:
jq --arg old "docker.org/alpha/alpha-app-newgen:12.2.3" --arg new "HELLO" -f update.jq input.json
update.jq:
walk(if type == "object" and .image == $old then .image=$new else . end)
If your jq does not have walk/1, then consider updating or getting its jq definition by googling: jq def walk
Targeted update
Invocation: as above
update.jq:
.taskDefinition.containerDefinitions[].image |= (if . == $old then $new else . end)
I'm pretty new to jq and wanted to use it to update an AWS ECS task definition with a new value. AWS cli returns the following json response and i would like to modify the object with name property CONFIG_URL with value "this is atest".
{
"family": "contentpublishing-task",
"volumes": [],
"containerDefinitions": [
{
"environment": [
{
"name": "TEST_ENV",
"value": "TEST"
},
{
"name": "CONFIG_URL",
"value": "s3://stg-appcfg/config-20160729-1130.json"
}
],
"name": "contentpublishing",
"mountPoints": [],
"image": "contentpublishing:blah",
"cpu": 512,
"portMappings": [
{
"protocol": "tcp",
"containerPort": 8081,
"hostPort": 8080
}
],
"memory": 256,
"essential": true,
"volumesFrom": []
}
]
}
Tried the following query
cat test.json | jq 'select(.containerDefinitions[0].environment[].name=="CONFIG_URL").value|="this is atest"' 2>&1
But following has been returned. As you can see an additional value key is added the outer most json object.
{
"family": "contentpublishing-task",
"volumes": [],
"containerDefinitions": [
{
"environment": [
{
"name": "TEST_ENV",
"value": "TEST"
},
{
"name": "CONFIG_URL",
"value": "s3://stg-appcfg/config-20160729-1130.json"
}
],
"name": "contentpublishing",
"mountPoints": [],
"image": "contentpublishing:blah",
"cpu": 512,
"portMappings": [
{
"protocol": "tcp",
"containerPort": 8081,
"hostPort": 8080
}
],
"memory": 256,
"essential": true,
"volumesFrom": []
}
],
"value": "this is atest"
}
You have to select the corresponding environment node first before setting the value. Your query doesn't change the context so it's still on the root item so you end up adding the new value to the root.
$ jq --arg update_name "CONFIG_URL" --arg update_value "this is a test" \
'(.containerDefinitions[].environment[] | select(.name == $update_name)).value = $update_value' input.json
Here is a solution which uses jq Complex assignments
(
.containerDefinitions[]
| .environment[]
| select(.name == "CONFIG_URL")
| .value
) |= "this is atest"