I have two json files.
I wanna merge objects in targetgroup.json and autoscaling.json with TargetGroupARNs and TargetGroupArn unique value in bash shell.
I am using jq version 1.4 and unable upgrade to last version because some cause.
#cat autoscaling.json
{
"AutoScalingGroups": [
{
"AutoScalingGroupARN": "arn:aws:autoscaling:ap-northeast-1:050073205187:autoScalingGroup:29f22791-b9cf-49c2-9ba6-b9ac2759d767:autoScalingGroupName/asg-dev-srv-01-20190327_133322",
"ServiceLinkedRoleARN": "arn:aws:iam::050073205187:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
"TargetGroupARNs": [
"arn:aws:elasticloadbalancing:ap-northeast-1:050073205187:targetgroup/dev-trial-tg/ef79ad5c9df9014e"
]
},
{
"AutoScalingGroupARN": "arn:aws:autoscaling:ap-northeast-1:050073205187:autoScalingGroup:cf662dc3-4e27-4022-a165-ebba8ca9488b:autoScalingGroupName/asg_group",
"ServiceLinkedRoleARN": "arn:aws:iam::050073205187:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
"TargetGroupARNs": [
"arn:aws:elasticloadbalancing:ap-northeast-1:050073205187:targetgroup/dev-trial1-tg/580ab71538a59063"
]
}
]
}
#cat targetgroup.json
[
{
"TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:050073205187:targetgroup/dev-trial-tg/ef79ad5c9df9014e",
"TargetGroupName": "dev-trial-tg"
},
{
"TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:050073205187:targetgroup/dev-trial1-tg/580ab71538a59063",
"TargetGroupName": "dev-trial1-tg"
}
]
My desired is:
[
{
"AutoScalingGroups": [
{
"AutoScalingGroupARN": "arn:aws:autoscaling:ap-northeast-1:050073205187:autoScalingGroup:29f22791-b9cf-49c2-9ba6-b9ac2759d767:autoScalingGroupName/asg-dev-srv-01-20190327_133322",
"ServiceLinkedRoleARN": "arn:aws:iam::050073205187:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
"TargetGroupARNs": [
"TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:050073205187:targetgroup/dev-trial-tg/ef79ad5c9df9014e",
"TargetGroupName": "dev-trial-tg"
],
},
{
"AutoScalingGroupARN": "arn:aws:autoscaling:ap-northeast-1:050073205187:autoScalingGroup:cf662dc3-4e27-4022-a165-ebba8ca9488b:autoScalingGroupName/asg_group",
"ServiceLinkedRoleARN": "arn:aws:iam::050073205187:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
"TargetGroupARNs": [
"TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:050073205187:targetgroup/dev-trial1-tg/580ab71538a59063",
"TargetGroupName": "dev-trial1-tg"
]
}
]
}
]
I tried, but it was too hard for me. I hope you can help me.
This is solution from #oguzismail
jq -n 'input | reduce (input|to_entries)[] as $p (.; .AutoScalingGroups[$p.key].TargetGroupARNs = $p.value)' autoscaling.json targetgroup.json
Related
I know this has to be simple, but for some reason it's eluding me how to find an element given a condition and modify one of its fields. The doc should be fully output (sed style) with the edit made.
{
"state": "wait",
"steps": {
"step1": [
{ "name":"Foo", "state":"wait" },
{ "name":"Bar", "state":"wait" }
],
"step2": [
{ "name":"Foo", "state":"wait" },
{ "name":"Zoinks", "state":"ready" }
],
"step3": [
{ "name":"Foo", "state":"cancel" }
]
}
}
I'm expecting something like this should be workable.
jq '. | (select(.steps[][].name=="Foo" and .steps[][].state=="wait") |= . + {.state:"Ready"}'
or
jq '. | (select(.steps[][]) | if (.name=="Foo" and .state=="wait") then (.state="Ready") else . end)
The desired output, of course, would be
{
"state": "wait",
"steps": {
"step1": [
{ "name":"Foo", "state":"ready" },
{ "name":"Bar", "state":"wait" }
],
"step2": [
{ "name":"Foo", "state":"ready" },
{ "name":"Zoinks", "state":"ready" }
],
"step3": [
{ "name":"Foo", "state":"cancel" }
]
}
}
Instead, when I'm not getting cryptic errors, I'm either modifying a top-level field in the document or modifying the field for all the elements or repeated the entire doc multiple times.
Any insights greatly appreciated.
Thanks.
p.s. is there a better syntax than [] to wildcard the named-elements under steps? Or after the pipe to identify the indices discovered by the select?
Pipe the output of .steps[][] into a select call that chooses the objects with the desired name and state values, then set the state value on the result.
$ jq '(.steps[][] | select(.name == "Foo" and .state == "wait")).state = "ready"' tmp.json
{
"state": "wait",
"steps": {
"step1": [
{
"name": "Foo",
"state": "ready"
},
{
"name": "Bar",
"state": "wait"
}
],
"step2": [
{
"name": "Foo",
"state": "ready"
},
{
"name": "Zoinks",
"state": "ready"
}
],
"step3": [
{
"name": "Foo",
"state": "cancel"
}
]
}
}
You can help confirm this using diff (the first jq just normalizes the formatting so that only the changes made by the second one show up in the diff):
$ diff <(jq . tmp.json) <(jq '...' tmp.json)
7c7
< "state": "wait"
---
> "state": "ready"
17c17
< "state": "wait"
---
> "state": "ready"
Is there a way to create a new element in an existing json object using jq? Example below:
Let's say I have this json object and would like to add a new element to foo:
json='{
"id": "<id>>",
"name": "<name>",
"properties": {
"State": "<state>",
"requests": [],
"foo": [
{
"id": "<id1>",
"bar1": [
{
"baz1": "*"
}
]
},
{
"id": "<id2>",
"bar2": [
{
"baz2": "*"
}
]
}
]
}
}'
This command works to do that:
json2=$($json1 | jq '.properties.foo += [ { "id": "<id3>", "bar3": [ { "baz3": "*"} ] } ]')
However, running that same command without a preexisting foo element fails (example array below):
json3='{
"id": "<id>>",
"name": "<name>",
"properties": {
"State": "<state>",
"requests": []
}
}'
Is there a way in jq to create that element in the json object if one already does not exist?
Thanks!
There is nothing wrong with your jq program, which can be seen by running:
jq '.properties.foo += [ { "id": "<id3>", "bar3": [ { "baz3": "*"} ] } ]' <<< "$json3"
It looks like the problem is with your invocation but since it's not clear what $json1 is, I'll just guess that the above is sufficient for you to resolve the issue.
I have a JSON like this (sample.json):
{
"sheet1": [
{
"hostname": "sv001",
"role": "web",
"ip1": "172.17.0.3"
},
{
"hostname": "sv002",
"role": "web",
"ip1": "172.17.0.4"
},
{
"hostname": "sv003",
"role": "db",
"ip1": "172.17.0.5",
"ip2": "172.18.0.5"
}
],
"sheet2": [
{
"hostname": "sv004",
"role": "web",
"ip1": "172.17.0.6"
},
{
"hostname": "sv005",
"role": "db",
"ip1": "172.17.0.7"
},
{
"hostname": "vsv006",
"role": "db",
"ip1": "172.17.0.8"
}
],
"sheet3": []
}
I want to extract data like this:
sheet1
jq '(something command)' sample.json
{
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
},
"db": {
"hosts": [
"172.17.0.5"
]
}
}
Is it possible to perform the reconstruction with jq map?
(I will reuse the result for ansible inventory.)
Here's a short, straight-forward and efficient solution -- efficient in part because it avoids group_by by courtesy of the following generic helper function:
def add_by(f;g): reduce .[] as $x ({}; .[$x|f] += [$x|g]);
.sheet1
| add_by(.role; .ip1)
| map_values( {hosts: .} )
Output
This produces the required output:
{
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
},
"db": {
"hosts": [
"172.17.0.5"
]
}
}
If the goal is to regroup the ips by their roles within each sheet you could do this:
map_values(
reduce group_by(.role)[] as $g ({};
.[$g[0].role].hosts = [$g[] | del(.hostname, .role)[]]
)
)
Which produces something like this:
{
"sheet1": {
"db": {
"hosts": [
"172.17.0.5",
"172.18.0.5"
]
},
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
}
},
"sheet2": {
"db": {
"hosts": [
"172.17.0.7",
"172.17.0.8"
]
},
"web": {
"hosts": [
"172.17.0.6"
]
}
},
"sheet3": {}
}
https://jqplay.org/s/3VpRc5l4_m
If you want to flatten all to a single object keeping only unique ips, you can keep everything mostly the same, you'll just need to flatten the inputs prior to grouping and remove the map_values/1 call.
$ jq -n '
reduce ([inputs[][]] | group_by(.role)[]) as $g ({};
.[$g[0].role].hosts = ([$g[] | del(.hostname, .role)[]] | unique)
)
'
{
"db": {
"hosts": [
"172.17.0.5",
"172.17.0.7",
"172.17.0.8",
"172.18.0.5"
]
},
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4",
"172.17.0.6"
]
}
}
https://jqplay.org/s/ZGj1wC8hU3
just want to get some expert tips on you on how to grab only this from AutoScalingGroups:
awseb-e-ASG
Actual JSON Output
{
"EnvironmentResources": {
"EnvironmentName": "MY-APP",
"AutoScalingGroups": [
{
"Name": "awseb-e-ASG"
}
],
"Triggers": [],
"LoadBalancers": [
{
"Name": "awseb-e-ELB"
}
],
"Queues": [],
"Instances": [
{
"Id": "i-XXXXXXXXXXXXXXXd"
}
],
"LaunchConfigurations": [
{
"Name": "awseb-e-LAUNCH"
}
]
}
}
I tried several commands but only getting this:
jq -r ".EnvironmentResources.LaunchConfigurations"
[
{
"Name": "awseb-e-ASG"
}
]
jq -r ".EnvironmentResources.LaunchConfigurations.Name"
jq: error: Cannot index array with string
awseb-e-ASG is under .AutoScalingGroups
so you could use the following filter:
.EnvironmentResources.AutoScalingGroups[].Name
Debugging
It usually pays to pay attention to the error message:
jq: error: Cannot index array with string
This, in effect, is telling you that .LaunchConfigurations.Name is erroneous because .LaunchConfigurations is an array, and therefore cannot have a string-valued key.
I have this file:
[
"smoke-tests",
"push-apps-manager"
]
I'd like to get this output using JQ:
{
"errands": [
{"name": "smoke-tests", "post_deploy": true},
{"name": "push-apps-manager", "post_deploy": true}
]
}
It seems so simple, yet, I have so much difficulty here...
It's a little tricky, since you need to embed the input into the list bound to the errands key. Start by creating the sequence of name/post_deploy objects:
% jq '.[] | {name: ., post_deploy: true}' names.json
{
"name": "smoke-tests",
"post_deploy": true
}
{
"name": "push-apps-manager",
"post_deploy": true
}
Then wrap that in the list in the outer object:
% jq '{errands: [.[] | {name: ., post_deploy: true}]}' names.json
{
"errands": [
{
"name": "smoke-tests",
"post_deploy": true
},
{
"name": "push-apps-manager",
"post_deploy": true
}
]
}
You can also use the map function (which I rarely remember how to use correctly, but it turns out is pretty simple here):
% jq '{errands: map({name:., post_deploy: true})}' names.json
Here is another approach. If you are new to jq it may be easiest to work towards the goal in small steps.
1) Start with the identity filter
.
which produces as expected
[
"smoke-tests",
"push-apps-manager"
]
2) next add the outer object with the "errands" key:
{ "errands": . }
which produces
{
"errands": [
"smoke-tests",
"push-apps-manager"
]
}
3) next move the data into an array
{ "errands": [ . ] }
which produces
{
"errands": [
[
"smoke-tests",
"push-apps-manager"
]
]
}
4) add the inner object with the "name" and "post_deploy" keys
{ "errands": [ { "name": ., "post_deploy": true } ] }
which produces
{
"errands": [
{
"name": [
"smoke-tests",
"push-apps-manager"
],
"post_deploy": true
}
]
}
5) Now we're really close. All we need to do is take advantage of jq's Object Construction behavior when an expression produces multiple results :
{ "errands": [ { "name": .[], "post_deploy": true } ] }
which gives us the desired result
{
"errands": [
{
"name": "smoke-tests",
"post_deploy": true
},
{
"name": "push-apps-manager",
"post_deploy": true
}
]
}