JQ: Merge all entries in an array without hardcoding the key - json

I have a json like this
[
{
"name": "hosts",
"ipaddress": "1.2.3.4",
"status": "UP",
"randomkey": "randomvalue"
},
{
"name": "hosts",
"ipaddress": "5.6.7.8",
"status": "DOWN",
"newkey": "newvalue"
},
{
"name": "hosts",
"ipaddress": "9.10.11.12",
"status": "RESTART",
"anotherkey": "anothervalue"
}
]
I want to merge the objects and looking for some output like this
[
{
"name": "hosts", //doesn't matter if it is ["hosts"]
"ipaddress": ["1.2.3.4", "5.6.7.8", "9.10.11.12"],
"status": ["UP", "DOWN", "RESTART"],
"randomkey": ["randomvalue"],
"newkey": ["newvalue"],
"anotherkey": ["anothervalue"]
}
]
I can hardcode each and every key and do something like this - { ipaddress: (map(.ipaddress) | unique ) } + { status: (map(.status) | unique ) } + { randomkey: (map(.randomkey) | unique ) }
The important ask here is the values are random and cannot be hardcoded.
Is there a way i can merge all the keys without hardcoding the key here?

Using reduce, then unique would be one way:
jq '[
reduce (.[] | to_entries[]) as {$key, $value} ({}; .[$key] += [$value])
| map_values(unique)
]'
[
{
"name": [
"hosts"
],
"ipaddress": [
"1.2.3.4",
"5.6.7.8",
"9.10.11.12"
],
"status": [
"DOWN",
"RESTART",
"UP"
],
"randomkey": [
"randomvalue"
],
"newkey": [
"newvalue"
],
"anotherkey": [
"anothervalue"
]
}
]
Demo
Using group_by and map, then unique again would be another:
jq '[
map(to_entries[]) | group_by(.key)
| map({key: first.key, value: map(.value) | unique})
| from_entries
]'
[
{
"anotherkey": [
"anothervalue"
],
"ipaddress": [
"1.2.3.4",
"5.6.7.8",
"9.10.11.12"
],
"name": [
"hosts"
],
"newkey": [
"newvalue"
],
"randomkey": [
"randomvalue"
],
"status": [
"DOWN",
"RESTART",
"UP"
]
}
]
Demo

Related

JQ - unique count of each value in an array

I'm needing to solve this with JQ. I have a large lists of arrays in my json file and am needing to do some sort | uniq -c types of stuff on them. Specifically I have a relatively nasty looking fruit array that needs to break down what is inside. I'm aware of unique and things like that, and imagine there is likely a simple way to do this, but I've been trying run down assigning things as variables and appending and whatnot, but I can't get the most basic part of counting the unique values per that fruit array, and especially not without breaking the rest of the content (hence the variable ideas). Please tell me I'm overthinking this.
I'd like to turn this;
[
{
"uid": "123abc",
"tID": [
"T19"
],
"fruit": [
"Kiwi",
"Apple",
"",
"",
"",
"Kiwi",
"",
"Kiwi",
"",
"",
"Mango",
"Kiwi"
]
},
{
"uid": "456xyz",
"tID": [
"T15"
],
"fruit": [
"",
"Orange"
]
}
]
Into this;
[
{
"uid": "123abc",
"tID": [
"T19"
],
"metadata": [
{
"name": "fruit",
"value": "Kiwi - 3"
},
{
"name": "fruit",
"value": "Mango - 1"
},
{
"name": "fruit",
"value": "Apple - 1"
}
]
},
{
"uid": "456xyz",
"tID": [
"T15"
],
"metadata": [
{
"name": "fruit",
"value": "Orange - 1"
}
]
}
]
Using group_by and length would be one way:
jq '
map(with_entries(select(.key == "fruit") |= (
.value |= (group_by(.) | map(
{name: "fruit", value: "\(.[0] | select(. != "")) - \(length)"}
))
| .key = "metadata"
)))
'
[
{
"uid": "123abc",
"tID": [
"T19"
],
"metadata": [
{
"name": "fruit",
"value": "Apple - 1"
},
{
"name": "fruit",
"value": "Kiwi - 4"
},
{
"name": "fruit",
"value": "Mango - 1"
}
]
},
{
"uid": "456xyz",
"tID": [
"T15"
],
"metadata": [
{
"name": "fruit",
"value": "Orange - 1"
}
]
}
]
Demo

How to get key value pairs of the objects from complex JSON using jq and map? (Active Campaign)

I have following JSON. I want to get key-value pair objects based on their role. In this example there are 3 roles(Presenter, Approver, Customer) but there can be more as it is dynamic.
JSON
{
"Presenter Name": "Roney",
"Presenter Email": "roney#domain.com",
"Approver Name": "Tim",
"Approver Email": "tim#domain.com",
"Customer Name": "Alex",
"Customer Email": "alex#domain.com",
"Invoice": "001",
"Date": "2022-02-14"
}
Expected output using jq, map,
{
"Presenter": {
"email_address": "roney#domain.com",
"name": "Roney",
"role": "Presenter"
},
"Approver": {
"email_address": "tim#domain.com",
"name": "Tim",
"role": "Approver"
},
"Customer": {
"email_address": "alex#domain.com",
"name": "Alex",
"role": "Customer"
}
}
I have tried till following but didn't get what to do next. Please advice.
to_entries |map( { (.key): { name: .value, email_address:.value, role: .key} } ) | add
This splits the keys at the space character while discarding any items that don't have one in it. Then it assigns the three fields to their values accordingly, using reduce to combine the grouping.
to_entries
| map(.key |= split(" ") | select(.key[1]))
| reduce group_by(.key[0])[] as $g ({};
.[$g[0].key[0]] = (
INDEX($g[]; .key[1]) | {
email_address: .Email.value,
name: .Name.value,
role: .Name.key[0]
}
)
)
{
"Approver": {
"email_address": "tim#domain.com",
"name": "Tim",
"role": "Approver"
},
"Customer": {
"email_address": "alex#domain.com",
"name": "Alex",
"role": "Customer"
},
"Presenter": {
"email_address": "roney#domain.com",
"name": "Roney",
"role": "Presenter"
}
}
Demo
Here's another, shorter approach that doesn't use group_by. Instead, this directly iterates over the initial object using reduce and imediately sets all the fields accordingly if the key followed the space-separated role-key pattern.
reduce (to_entries[] | .key /= " ") as {key: [$role, $key], $value} ({};
if $key then
.[$role] += {({Email: "email_address", Name: "name"}[$key]): $value, $role}
else . end
)
{
"Presenter": {
"name": "Roney",
"role": "Presenter",
"email_address": "roney#domain.com"
},
"Approver": {
"name": "Tim",
"role": "Approver",
"email_address": "tim#domain.com"
},
"Customer": {
"name": "Alex",
"role": "Customer",
"email_address": "alex#domain.com"
}
}
Demo
{ "Name": "name", "Email": "email_address" } as $key_map |
to_entries |
map (
( .key | split(" ") | select( length == 2 ) ) as [ $role, $raw_key ] |
[ $role, "role", $role ],
[ $role, $key_map[$raw_key], .value ]
) |
reduce .[] as [ $role, $key, $val ] ( {}; .[ $role ][ $key ] = $val )
Demo on jqplay
In the above, we start by making the data uniform. Specifically, we start by producing the following:
[
[ "Presenter", "role", "Presenter" ],
[ "Presenter", "name", "Roney" ],
[ "Presenter", "role", "Presenter" ],
[ "Presenter", "email_address", "roney#domain.com" ],
[ "Approver", "role", "Approver" ],
[ "Approver", "name", "Tim" ],
[ "Approver", "role", "Approver" ],
[ "Approver", "email_address", "tim#domain.com" ],
[ "Customer", "role", "Customer" ],
[ "Customer", "name", "Alex" ],
[ "Customer", "role", "Customer" ],
[ "Customer", "email_address", "alex#domain.com" ]
]
There's redundant information, but that doesn't matter.
Then, the final simple reduce builds the desired structure.
.key | split(" ") | select( length == 2 )
can be replaced with the safer
.key | match("^(.*) (Name|Email)$") | .captures | map( .string )

jq: error (at <stdin>:125): Cannot index array with string "DefId"

I have below json data. Need help in parsing it.
{
"e4624072-a9a2-4181-9649-550b9cfeb7cd||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "A1",
"data1": 1073741824,
"data2": 0,
"data3": 0
}
]
}
],
"name": [
"AMIT||220000d7f728801"
],
"id": "e4624072-a9a2-4181-9649-550b9cfeb7cd||220000d7f738801",
"endTime": 96285337200000
},
"0b1141b2-c2de-47c9-aa0c-2742f92b63f2||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "B1",
"data1": 5368709120,
"data2": 5368709120,
"data3": 5368709120
}
]
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "BoostYES_B1"
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "BoostOff_B1"
}
],
"name": [
"AMIT||220000d7f728801"
],
"id": "0b1141b2-c2de-47c9-aa0c-2742f92b63f2||220000d7f738801",
"lifeCycle": 0
},
"23e529f9-b2f3-4730-9b28-4ee05ca678b6||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "A2",
"data1": 1073741824,
"data2": 0,
"data3": 0,
"lastUpdateTime": 1619541451476,
"origInitialVal": 1073741824
}
]
}
],
"name": [
"AMIT||220000d7f728802"
],
"id": "23e529f9-b2f3-4730-9b28-4ee05ca678b6||220000d7f738801",
"endTime": 96285337200000,
"lifeCycle": 0
},
"66b2229b-2c16-4d54-b2a8-fcc1baeaf51c||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "B2",
"data1": 10737418240,
"data2": 10737418240,
"data3": 10737418240,
"lastUpdateTime": 1637766239807,
"origInitialVal": 10737418240
}
]
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "RMN_B2"
}
],
"name": [
"AMIT||220000d7f728801"
],
"startTime": 1637766000000,
"id": "66b2229b-2c16-4d54-b2a8-fcc1baeaf51c||220000d7f738801",
"endTime": 1637852400000,
"lifeCycle": 0
},
"b896eb1b-d6b0-432b-8925-af17431c0f3e||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "B3",
"data1": 2147483648,
"data2": 2147483648,
"data3": 2147483648,
"lastUpdateTime": 1635692405780
}
]
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "BoostYES_B3"
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "BoostOffCS_B3"
}
],
"name": [
"AMIT||220000d7f728801"
],
"id": "b896eb1b-d6b0-432b-8925-af17431c0f3e||220000d7f738801",
"lifeCycle": 0
}
}
Required output is as below :
AMIT||220000d7f728801,A1,1073741824,0,0
AMIT||220000d7f728801,B1,5368709120,5368709120,5368709120
AMIT||220000d7f728801,A2,1073741824,0,0
AMIT||220000d7f728801,B2,10737418240,10737418240,10737418240
AMIT||220000d7f728801,B3,2147483648,2147483648,2147483648
I tried to execute below jq to start with but it giving as mentioned in title. This might be due to DefId is not present at all the places. Similarly for data 1, data2 and data3.
jq -r '.[] | ."name", ."List"[]."Insts"."DefId"'
What is the right jq command in order to get the correct output?
Unfortunately you say nothing about constraints (guaranteed existence of fields, lengths of arrays, etc.) regarding your data structure. Therefore, simply based on what you have provided, this will extract the contents into your desired output format:
jq -r '.[] | [.name[0] , (.List[0].Insts[0] | .DefId, .data1, .data2, .data3)] | join(",")'
AMIT||220000d7f728801,A1,1073741824,0,0
AMIT||220000d7f728801,B1,5368709120,5368709120,5368709120
AMIT||220000d7f728801,A2,1073741824,0,0
AMIT||220000d7f728801,B2,10737418240,10737418240,10737418240
AMIT||220000d7f728801,B3,2147483648,2147483648,2147483648
Demo
You can dynamically walk within List.Insts array without error by using
.List[].Insts | select( . != null ) for the returning values of to_entries[]
while it's straightforward to extract the value for the name key.
Then join the components of the formed array through use of wrapper square brackets such as
jq -r 'to_entries[] | .value | [ .name[] , (.List[].Insts | select( . != null ) | .[] | .DefId, "\(.data1)", "\(.data2)", "\(.data3)" ) ] | join(",")'
Demo

Extract from json with | jq by a given word

Can somebody help me to extract with | jq the following:
{
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {
"pod": "dev-cds-5c97cf7f78-sw6b9"
},
"values": [
[
1588204800,
"0.3561394483796914"
],
[
1588215600,
"0.3607968456046861"
],
[
1588226400,
"0.3813882532417868"
],
[
1588237200,
"0.6264355815408573"
]
]
},
{
"metric": {
"pod": "uat-cds-66ccc9685-b5tvh"
},
"values": [
[
1588204800,
"0.9969746974696218"
],
[
1588215600,
"0.7400881057270005"
],
[
1588226400,
"1.2298959318837195"
],
[
1588237200,
"0.9482296838254507"
]
]
}
]
}
}
I need to obtain all-values individually by given word dev-cds and not all the name dev-cds-5c97cf7f78-sw6b9.
Result desired:
{
"metric": {
"pod": "dev-cds-5c97cf7f78-sw6b9"
},
"values": [
[
1588204800,
"0.3561394483796914"
],
[
1588215600,
"0.3607968456046861"
],
[
1588226400,
"0.3813882532417868"
],
[
1588237200,
"0.6264355815408573"
]
]
}
You should first iterate over the result array. Check if the pod inside, metric object has the value that contains "dev-cds".
.data.result[] | if .metric.pod | contains("dev-cds") then . else empty end
https://jqplay.org/s/54OH83qHKP

Using jq to find and replace elements from Json sub collection

I am trying to parse convert Json Task definition with new image.
I want to change the "image": "docker.org/alpha/alpha-app-newgen:12.2.3" value in Json to "image": "docker.org/alpha/alpha-app-newgen:12.5.0" or any other version dynamically.
below is my Json task def:
{
"taskDefinition": {
"family": "ing-stack",
"volumes": [
{
"host": {
"sourcePath": "/tmp/nginx/elb.conf"
},
"name": "volume-0"
}
],
"containerDefinitions": [
{
"dnsSearchDomains": [],
"environment": [
{
"name": "API_SECRET",
"value": "ING-SECRET"
},
{
"name": "API_KEY",
"value": "AVERA-CADA-VERA-KEY"
}
],
"readonlyRootFilesystem": false,
"name": "ing-stg",
"links": [],
"mountPoints": [],
"image": "docker.org/alpha/alpha-app-newgen:12.2.3",
"privileged": false,
"essential": true,
"portMappings": [
{
"protocol": "tcp",
"containerPort": 19000,
"hostPort": 19000
}
],
"dockerLabels": {}
},
{
"dnsSearchDomains": [],
"environment": [
{
"name": "NG_PROXY",
"value": "ing"
}
],
"readonlyRootFilesystem": false,
"name": "web",
"links": [
"identity-ng"
],
"mountPoints": [
{
"sourceVolume": "volume-0",
"readOnly": false,
"containerPath": "/etc/nginx/conf.d/default.conf"
}
],
"image": "docker.org/alpha/alpha-ui:6.4.7",
"portMappings": [
{
"protocol": "tcp",
"containerPort": 443,
"hostPort": 443
},
{
"protocol": "tcp",
"containerPort": 80,
"hostPort": 80
}
],
"memory": 512,
"command": [
"sh",
"prep-run-nginx.sh"
],
"dockerLabels": {}
}
],
"revision": 136
}
}
I need to get the same structure back with new value for the image.
I tried the following
jq '. | select(.containerDefinitions[].image | contains("'$new_img_no_ver'") ) | .image |= "my new image"', but its adding to the end of the JSON.
Can anybody tell me how to achieve this.
Here are two potential solutions. Other variants are of course possible.
walk/1
If you don't want to be bothered with the details of exactly where the relevant "image" tag is located, consider using walk/1:
Invocation:
jq --arg old "docker.org/alpha/alpha-app-newgen:12.2.3" --arg new "HELLO" -f update.jq input.json
update.jq:
walk(if type == "object" and .image == $old then .image=$new else . end)
If your jq does not have walk/1, then consider updating or getting its jq definition by googling: jq def walk
Targeted update
Invocation: as above
update.jq:
.taskDefinition.containerDefinitions[].image |= (if . == $old then $new else . end)