Merging two files using jq - json

I've got two files.
First:
[
{ "person1": [] },
{ "person2": [] }
]
Second:
[
{
"person2": { "attribute1": "wer", "attribute2": "sdf" }
},
{
"person2": { "attribute1": "ert", "attribute2": "dfg" }
},
{
"person2": { "attribute1": "rty", "attribute2": "fgh" }
},
{
"person3": { "attribute1": "tyu", "attribute2": "ghj" }
},
{
"person1": { "attribute1": "yui", "attribute2": "hjk" }
}
]
I try to merge them, using jq. For each person from the first file (in the second file migth be more persons, which should be ignored) create list of it's attributes. So an output should look something like this:
[
{
"person1":
[
{ "attribute1": "yui", "attribute2": "hjk" }
]
},
{
"person2":
[
{ "attribute1": "wer", "attribute2": "sdf" },
{ "attribute1": "ert", "attribute2": "dfg" },
{ "attribute1": "rty", "attribute2": "fgh" }
]
}
]
I tried different options, but I can't achieve expected result.

jq 'reduce (input[]|to_entries[]) as $e (add;
if has($e.key) then .[$e.key] += [$e.value] else . end
) | [keys_unsorted[] as $k|{($k): .[$k]}]' file1 file2
online demo

The following solution is focused on efficiency, and also suggests an alternative format for the output:
jq program (merge.jq)
add as $dict
# aggregation:
| (reduce input[] as $record ({};
($record|keys_unsorted[0]) as $person
| if $dict[$person] then .[$person] += [$record[$person]] else . end )) as $answer
# re-arrangement
| reduce ($dict|keys_unsorted[]) as $person ([]; . + [ {($person): $answer[$person] } ] )
Invocation
jq -f merge.jq first.json second.json

Related

jq - remove non-matching fields in "object-array-with-objects"

Given the following JSON-object:
{
"meta": {
"data1": {
"keep": { "key": "value" }
}
},
"detail": {
"data2": [
{
"keep1": "keep1value",
"keep2": "keep2value",
"nokeep1": "abc"
}
],
"data3": [
{
"keep1": "keep1value",
"keep2": "keep2value",
"nokeep2": { "abc": "def" }
}
]
},
"drop" : "this"
}
I'm trying to clean it by removing unwanted fields, like "remove", "nokeep1" and "nokeep2".
However objects in the "data2" and "data3" arrays might contain more fields than the example "nokeepX", but will always contain "keep1" and "keep2" which I want to keep.
My desired output is the following JSON:
{
"meta": { "data1": { "keep": { "key": "value" } } },
"detail": {
"data2": [
{
"keep1": "keep1value",
"keep2": "keep2value"
}
],
"data3": [
{
"keep1": "keep1value",
"keep2": "keep2value"
}
]
}
}
I've managed to remove the "drop" field with this query:
jq 'def pick($paths): . as $root | reduce ($paths[]|[.]|flatten(1)) as $path ({}; . + setpath($path; $root|getpath($path))); pick([["meta"], ["detail", "data2"], ["detail", "data3"]])'
However I've been struggling to figure out how to remove the "nokeepX" fields - is it possible to accomplish this?
If you have only a limited set of properties, it could be easier not to remove unwanted fields, but create the output from the required fields only:
{
meta,
detail: .detail | {
data2: .data2 | map({ keep1, keep2 }),
data3: .data3 | map({ keep1, keep2 })
}
}
Output:
{
"meta": {
"data1": {
"keep": {
"key": "value"
}
}
},
"detail": {
"data2": [
{
"keep1": "keep1value",
"keep2": "keep2value"
}
],
"data3": [
{
"keep1": "keep1value",
"keep2": "keep2value"
}
]
}
}
The approach can be combined with dropping certain fields:
{
meta,
detail: .detail | {
data2: .data2 | map(del(.nokeep1)),
data3: .data3 | map(del(.nokeep2))
}
}
producing the same output as above.
Just provide all the concrete paths to del:
del(
.detail.data2[0].nokeep1,
.detail.data3[0].nokeep2,
.drop
)
Demo
Or generalize by e.g. traversing all array items (not just the first) using [] without indices:
del(
.detail.data2[].nokeep1,
.detail.data3[].nokeep2,
.drop
)
Demo
Or go arbitrarily deep using .., and just provide the deepest field names:
del(.. | objects | .nokeep1, .nokeep2, .drop)
Demo
Output:
{
"meta": {
"data1": {
"keep": "true"
}
},
"detail": {
"data2": [
{
"keep1": "keep1value",
"keep2": "keep2value"
}
],
"data3": [
{
"keep1": "keep1value",
"keep2": "keep2value"
}
]
}
}
For the other way round, you could list all the leaf paths using paths(scalars), filter out those where the deepest level .[-1] does not match your criteria, and use delpaths to remove the remaining leafs:
delpaths([paths(scalars) | select(
.[-1] | IN("keep", "keep1", "keep2") | not
)])
Demo

Recursively replace value with $path in JQ

Assuming that i have a complex JsonObject
{
"parent": {
"name": "value",
"child": {
"child_value1": "value",
"child_value2": "value",
"child_value3": ["value1","value2"],
"child_value4": {
"child_child_value1":"value"
}
}
}
}
I want to replace the value of each key, with the name of key prefixed with $
{
"parent": {
"name": "$name",
"child": {
"child_value1": "$child_child_value1",
"child_value2": "$child_child_value2",
"child_value3": ["$child_child_value3_0","$child_child_value3_1"],
"child_value4": {
"child_child_value1":"$child_child_value4_child_child_value1"
}
}
}
}
Is there a way to do it recursively?
EDIT
This is the currently configuration file that I am using
{
"apis": {
"order": {
"base_url": "$mapping_base_url"
},
"payment": {
"base_url": "$admin_base_url"
}
},
"features": {
"authentication": {
"authProviders": true,
"registration": false
}
},
"availableLocales":["en","es"]
}
This is the result using the recomended jq expression:
. |= reduce paths(strings) as $p (.; setpath($p; "$" + ($p | join("_"))))
{
"apis": {
"order": {
"base_url": "$apis_order_base_url"
},
"payment": {
"base_url": "$apis_payment_base_url"
}
},
"features": {
"authentication": {
"authProviders": true,
"registration": false
}
},
"availableLocales": [
"$availableLocales_0",
"$availableLocales_1"
]
}
You're looking for something like this:
.parent |=
reduce paths(strings) as $p (.;
setpath($p; "$" + ($p | join("_")))
)
Online demo

Reconstructing JSON with jq

I have a JSON like this (sample.json):
{
"sheet1": [
{
"hostname": "sv001",
"role": "web",
"ip1": "172.17.0.3"
},
{
"hostname": "sv002",
"role": "web",
"ip1": "172.17.0.4"
},
{
"hostname": "sv003",
"role": "db",
"ip1": "172.17.0.5",
"ip2": "172.18.0.5"
}
],
"sheet2": [
{
"hostname": "sv004",
"role": "web",
"ip1": "172.17.0.6"
},
{
"hostname": "sv005",
"role": "db",
"ip1": "172.17.0.7"
},
{
"hostname": "vsv006",
"role": "db",
"ip1": "172.17.0.8"
}
],
"sheet3": []
}
I want to extract data like this:
sheet1
jq '(something command)' sample.json
{
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
},
"db": {
"hosts": [
"172.17.0.5"
]
}
}
Is it possible to perform the reconstruction with jq map?
(I will reuse the result for ansible inventory.)
Here's a short, straight-forward and efficient solution -- efficient in part because it avoids group_by by courtesy of the following generic helper function:
def add_by(f;g): reduce .[] as $x ({}; .[$x|f] += [$x|g]);
.sheet1
| add_by(.role; .ip1)
| map_values( {hosts: .} )
Output
This produces the required output:
{
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
},
"db": {
"hosts": [
"172.17.0.5"
]
}
}
If the goal is to regroup the ips by their roles within each sheet you could do this:
map_values(
reduce group_by(.role)[] as $g ({};
.[$g[0].role].hosts = [$g[] | del(.hostname, .role)[]]
)
)
Which produces something like this:
{
"sheet1": {
"db": {
"hosts": [
"172.17.0.5",
"172.18.0.5"
]
},
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
}
},
"sheet2": {
"db": {
"hosts": [
"172.17.0.7",
"172.17.0.8"
]
},
"web": {
"hosts": [
"172.17.0.6"
]
}
},
"sheet3": {}
}
https://jqplay.org/s/3VpRc5l4_m
If you want to flatten all to a single object keeping only unique ips, you can keep everything mostly the same, you'll just need to flatten the inputs prior to grouping and remove the map_values/1 call.
$ jq -n '
reduce ([inputs[][]] | group_by(.role)[]) as $g ({};
.[$g[0].role].hosts = ([$g[] | del(.hostname, .role)[]] | unique)
)
'
{
"db": {
"hosts": [
"172.17.0.5",
"172.17.0.7",
"172.17.0.8",
"172.18.0.5"
]
},
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4",
"172.17.0.6"
]
}
}
https://jqplay.org/s/ZGj1wC8hU3

How to merge object TargetGroupARNs and TargetGroupArn unique in bash shell?

I have two json files.
I wanna merge objects in targetgroup.json and autoscaling.json with TargetGroupARNs and TargetGroupArn unique value in bash shell.
I am using jq version 1.4 and unable upgrade to last version because some cause.
#cat autoscaling.json
{
"AutoScalingGroups": [
{
"AutoScalingGroupARN": "arn:aws:autoscaling:ap-northeast-1:050073205187:autoScalingGroup:29f22791-b9cf-49c2-9ba6-b9ac2759d767:autoScalingGroupName/asg-dev-srv-01-20190327_133322",
"ServiceLinkedRoleARN": "arn:aws:iam::050073205187:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
"TargetGroupARNs": [
"arn:aws:elasticloadbalancing:ap-northeast-1:050073205187:targetgroup/dev-trial-tg/ef79ad5c9df9014e"
]
},
{
"AutoScalingGroupARN": "arn:aws:autoscaling:ap-northeast-1:050073205187:autoScalingGroup:cf662dc3-4e27-4022-a165-ebba8ca9488b:autoScalingGroupName/asg_group",
"ServiceLinkedRoleARN": "arn:aws:iam::050073205187:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
"TargetGroupARNs": [
"arn:aws:elasticloadbalancing:ap-northeast-1:050073205187:targetgroup/dev-trial1-tg/580ab71538a59063"
]
}
]
}
#cat targetgroup.json
[
{
"TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:050073205187:targetgroup/dev-trial-tg/ef79ad5c9df9014e",
"TargetGroupName": "dev-trial-tg"
},
{
"TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:050073205187:targetgroup/dev-trial1-tg/580ab71538a59063",
"TargetGroupName": "dev-trial1-tg"
}
]
My desired is:
[
{
"AutoScalingGroups": [
{
"AutoScalingGroupARN": "arn:aws:autoscaling:ap-northeast-1:050073205187:autoScalingGroup:29f22791-b9cf-49c2-9ba6-b9ac2759d767:autoScalingGroupName/asg-dev-srv-01-20190327_133322",
"ServiceLinkedRoleARN": "arn:aws:iam::050073205187:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
"TargetGroupARNs": [
"TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:050073205187:targetgroup/dev-trial-tg/ef79ad5c9df9014e",
"TargetGroupName": "dev-trial-tg"
],
},
{
"AutoScalingGroupARN": "arn:aws:autoscaling:ap-northeast-1:050073205187:autoScalingGroup:cf662dc3-4e27-4022-a165-ebba8ca9488b:autoScalingGroupName/asg_group",
"ServiceLinkedRoleARN": "arn:aws:iam::050073205187:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
"TargetGroupARNs": [
"TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:050073205187:targetgroup/dev-trial1-tg/580ab71538a59063",
"TargetGroupName": "dev-trial1-tg"
]
}
]
}
]
I tried, but it was too hard for me. I hope you can help me.
This is solution from #oguzismail
jq -n 'input | reduce (input|to_entries)[] as $p (.; .AutoScalingGroups[$p.key].TargetGroupARNs = $p.value)' autoscaling.json targetgroup.json

Create one JSON from another, with extra data va JQ

I have this file:
[
"smoke-tests",
"push-apps-manager"
]
I'd like to get this output using JQ:
{
"errands": [
{"name": "smoke-tests", "post_deploy": true},
{"name": "push-apps-manager", "post_deploy": true}
]
}
It seems so simple, yet, I have so much difficulty here...
It's a little tricky, since you need to embed the input into the list bound to the errands key. Start by creating the sequence of name/post_deploy objects:
% jq '.[] | {name: ., post_deploy: true}' names.json
{
"name": "smoke-tests",
"post_deploy": true
}
{
"name": "push-apps-manager",
"post_deploy": true
}
Then wrap that in the list in the outer object:
% jq '{errands: [.[] | {name: ., post_deploy: true}]}' names.json
{
"errands": [
{
"name": "smoke-tests",
"post_deploy": true
},
{
"name": "push-apps-manager",
"post_deploy": true
}
]
}
You can also use the map function (which I rarely remember how to use correctly, but it turns out is pretty simple here):
% jq '{errands: map({name:., post_deploy: true})}' names.json
Here is another approach. If you are new to jq it may be easiest to work towards the goal in small steps.
1) Start with the identity filter
.
which produces as expected
[
"smoke-tests",
"push-apps-manager"
]
2) next add the outer object with the "errands" key:
{ "errands": . }
which produces
{
"errands": [
"smoke-tests",
"push-apps-manager"
]
}
3) next move the data into an array
{ "errands": [ . ] }
which produces
{
"errands": [
[
"smoke-tests",
"push-apps-manager"
]
]
}
4) add the inner object with the "name" and "post_deploy" keys
{ "errands": [ { "name": ., "post_deploy": true } ] }
which produces
{
"errands": [
{
"name": [
"smoke-tests",
"push-apps-manager"
],
"post_deploy": true
}
]
}
5) Now we're really close. All we need to do is take advantage of jq's Object Construction behavior when an expression produces multiple results :
{ "errands": [ { "name": .[], "post_deploy": true } ] }
which gives us the desired result
{
"errands": [
{
"name": "smoke-tests",
"post_deploy": true
},
{
"name": "push-apps-manager",
"post_deploy": true
}
]
}