JQ: group input and and generate output json - json

Consider the below input
{
"name": "examplename1",
"Date1": "value1",
"Date2": "value2",
"Date3": "value3"
}
{
"name": "examplename1",
"Date1": "value4",
"Date2": "value5",
"Date3": "value6"
}
{
"name": "examplename2",
"Date1": "value7",
"Date2": "value8",
"Date3": "value9"
}
{
"name": "examplename2",
"Date1": "value10",
"Date2":"value11",
"Date3": "value12"
}
Require output as below
{
"names": "examplename1",
"availabledates1":[
"value1",
"value4"
],
"availabledates2":[
"value2",
"value5"
],
"availabledates3":[
"value3",
"value6"
]
}
{
"names": "examplename2",
"availabledates1":[
"value7",
"value10"
],
"availabledates2":[
"value8",
"valu11"
],
"availabledates3":[
"value9",
"value12"
]
}
Using JQ
[inputs] | group_by(.name)[] | [{names: .[].name, availabledates1: [.[].Date1], availabledates2: [.[].Date2], availabledates3: [.[].Date3]}] | unique_by(.names) | .[]
Getting output
{
"names": "examplename1",
"availabledates1": [
"value4"
],
"availabledates2": [
"value5"
],
"availabledates3": [
"value6"
]
}
{
"names": "examplename2",
"availabledates1": [
"value7",
"value10"
],
"availabledates2": [
"value8",
"value11"
],
"availabledates3": [
"value9",
"value12"
]
}
Issue 1: This JQ ignores the first row in inputs.
Issue 2: If the input data set is very large this jq takes too much memory and eventually fails to execute as its doing multiple iterations which needs parallel threads.
Can refer this: https://jqplay.org/s/OOHAuv72GAL
Need here more efficient jq which does not fail on large data set and also considers first row in inputs.

Using group_by:
jq -n '
[inputs] | group_by(.name)[] | {
names: first.name,
avaliableDates1: map(.Date1),
avaliableDates2: map(.Date2),
avaliableDates3: map(.Date3)
}
'
Demo
Using reduce:
jq -n '
(reduce inputs as $i ({}; .[$i.name] |= (
.names = $i.name
| .avaliableDates1 += [$i.Date1]
| .avaliableDates2 += [$i.Date2]
| .avaliableDates3 += [$i.Date3]
)))[]
'
Demo
Output:
{
"names": "examplename1",
"avaliableDates1": [
"value1",
"value4"
],
"avaliableDates2": [
"value2",
"value5"
],
"avaliableDates3": [
"value3",
"value6"
]
}
{
"names": "examplename2",
"avaliableDates1": [
"value7",
"value10"
],
"avaliableDates2": [
"value8",
"value11"
],
"avaliableDates3": [
"value9",
"value12"
]
}

Related

JQ: Merge all entries in an array without hardcoding the key

I have a json like this
[
{
"name": "hosts",
"ipaddress": "1.2.3.4",
"status": "UP",
"randomkey": "randomvalue"
},
{
"name": "hosts",
"ipaddress": "5.6.7.8",
"status": "DOWN",
"newkey": "newvalue"
},
{
"name": "hosts",
"ipaddress": "9.10.11.12",
"status": "RESTART",
"anotherkey": "anothervalue"
}
]
I want to merge the objects and looking for some output like this
[
{
"name": "hosts", //doesn't matter if it is ["hosts"]
"ipaddress": ["1.2.3.4", "5.6.7.8", "9.10.11.12"],
"status": ["UP", "DOWN", "RESTART"],
"randomkey": ["randomvalue"],
"newkey": ["newvalue"],
"anotherkey": ["anothervalue"]
}
]
I can hardcode each and every key and do something like this - { ipaddress: (map(.ipaddress) | unique ) } + { status: (map(.status) | unique ) } + { randomkey: (map(.randomkey) | unique ) }
The important ask here is the values are random and cannot be hardcoded.
Is there a way i can merge all the keys without hardcoding the key here?
Using reduce, then unique would be one way:
jq '[
reduce (.[] | to_entries[]) as {$key, $value} ({}; .[$key] += [$value])
| map_values(unique)
]'
[
{
"name": [
"hosts"
],
"ipaddress": [
"1.2.3.4",
"5.6.7.8",
"9.10.11.12"
],
"status": [
"DOWN",
"RESTART",
"UP"
],
"randomkey": [
"randomvalue"
],
"newkey": [
"newvalue"
],
"anotherkey": [
"anothervalue"
]
}
]
Demo
Using group_by and map, then unique again would be another:
jq '[
map(to_entries[]) | group_by(.key)
| map({key: first.key, value: map(.value) | unique})
| from_entries
]'
[
{
"anotherkey": [
"anothervalue"
],
"ipaddress": [
"1.2.3.4",
"5.6.7.8",
"9.10.11.12"
],
"name": [
"hosts"
],
"newkey": [
"newvalue"
],
"randomkey": [
"randomvalue"
],
"status": [
"DOWN",
"RESTART",
"UP"
]
}
]
Demo

jq: error (at <stdin>:125): Cannot index array with string "DefId"

I have below json data. Need help in parsing it.
{
"e4624072-a9a2-4181-9649-550b9cfeb7cd||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "A1",
"data1": 1073741824,
"data2": 0,
"data3": 0
}
]
}
],
"name": [
"AMIT||220000d7f728801"
],
"id": "e4624072-a9a2-4181-9649-550b9cfeb7cd||220000d7f738801",
"endTime": 96285337200000
},
"0b1141b2-c2de-47c9-aa0c-2742f92b63f2||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "B1",
"data1": 5368709120,
"data2": 5368709120,
"data3": 5368709120
}
]
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "BoostYES_B1"
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "BoostOff_B1"
}
],
"name": [
"AMIT||220000d7f728801"
],
"id": "0b1141b2-c2de-47c9-aa0c-2742f92b63f2||220000d7f738801",
"lifeCycle": 0
},
"23e529f9-b2f3-4730-9b28-4ee05ca678b6||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "A2",
"data1": 1073741824,
"data2": 0,
"data3": 0,
"lastUpdateTime": 1619541451476,
"origInitialVal": 1073741824
}
]
}
],
"name": [
"AMIT||220000d7f728802"
],
"id": "23e529f9-b2f3-4730-9b28-4ee05ca678b6||220000d7f738801",
"endTime": 96285337200000,
"lifeCycle": 0
},
"66b2229b-2c16-4d54-b2a8-fcc1baeaf51c||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "B2",
"data1": 10737418240,
"data2": 10737418240,
"data3": 10737418240,
"lastUpdateTime": 1637766239807,
"origInitialVal": 10737418240
}
]
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "RMN_B2"
}
],
"name": [
"AMIT||220000d7f728801"
],
"startTime": 1637766000000,
"id": "66b2229b-2c16-4d54-b2a8-fcc1baeaf51c||220000d7f738801",
"endTime": 1637852400000,
"lifeCycle": 0
},
"b896eb1b-d6b0-432b-8925-af17431c0f3e||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "B3",
"data1": 2147483648,
"data2": 2147483648,
"data3": 2147483648,
"lastUpdateTime": 1635692405780
}
]
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "BoostYES_B3"
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "BoostOffCS_B3"
}
],
"name": [
"AMIT||220000d7f728801"
],
"id": "b896eb1b-d6b0-432b-8925-af17431c0f3e||220000d7f738801",
"lifeCycle": 0
}
}
Required output is as below :
AMIT||220000d7f728801,A1,1073741824,0,0
AMIT||220000d7f728801,B1,5368709120,5368709120,5368709120
AMIT||220000d7f728801,A2,1073741824,0,0
AMIT||220000d7f728801,B2,10737418240,10737418240,10737418240
AMIT||220000d7f728801,B3,2147483648,2147483648,2147483648
I tried to execute below jq to start with but it giving as mentioned in title. This might be due to DefId is not present at all the places. Similarly for data 1, data2 and data3.
jq -r '.[] | ."name", ."List"[]."Insts"."DefId"'
What is the right jq command in order to get the correct output?
Unfortunately you say nothing about constraints (guaranteed existence of fields, lengths of arrays, etc.) regarding your data structure. Therefore, simply based on what you have provided, this will extract the contents into your desired output format:
jq -r '.[] | [.name[0] , (.List[0].Insts[0] | .DefId, .data1, .data2, .data3)] | join(",")'
AMIT||220000d7f728801,A1,1073741824,0,0
AMIT||220000d7f728801,B1,5368709120,5368709120,5368709120
AMIT||220000d7f728801,A2,1073741824,0,0
AMIT||220000d7f728801,B2,10737418240,10737418240,10737418240
AMIT||220000d7f728801,B3,2147483648,2147483648,2147483648
Demo
You can dynamically walk within List.Insts array without error by using
.List[].Insts | select( . != null ) for the returning values of to_entries[]
while it's straightforward to extract the value for the name key.
Then join the components of the formed array through use of wrapper square brackets such as
jq -r 'to_entries[] | .value | [ .name[] , (.List[].Insts | select( . != null ) | .[] | .DefId, "\(.data1)", "\(.data2)", "\(.data3)" ) ] | join(",")'
Demo

Merge multiple objects under a single object and add sibling object in one command

So I have three files:
cats.json
{
"cats": [
{
"name": "fluffles",
"age": 10,
"color": "white"
}
]
}
dogs.json
{
"dogs": [
{
"name": "sam",
"age": 5,
"color": "black and white"
},
{
"name": "rover",
"age": 2,
"color": "brown and white"
}
]
}
snakes.json
{
"snakes": [
{
"name": "noodles",
"age": 10,
"color": "green"
}
]
}
I wanted to merge these together, under an "animals" object. I've found that will merge the files:
jq -s '{"animals": .} ' cats.json dogs.json snakes.json > animals.json
{
"animals": [
{
"cats": [
{
"name": "fluffles",
"age": 10,
"color": "white"
}
]
},
{
"dogs": [
{
"name": "sam",
"age": 5,
"color": "black and white"
},
{
"name": "rover",
"age": 2,
"color": "brown and white"
}
]
},
{
"snakes": [
{
"name": "noodles",
"age": 10,
"color": "green"
}
]
}
]
}
Now I have an additional object:
owners.json
{
"owners": [
"peter",
"william",
"sally"
]
}
which I want to merge into the same file using
jq -s '.[0] + .[1]' animals.json owners.json
Can I do both of these operations with just one jq command?
jq -s '{"animals": .} ' cats.json dogs.json snakes.json > animals.json
jq -s '.[0] + .[1]' animals.json owners.json
The result would look like this:
{
"animals": [
{
"cats": [
{
"name": "fluffles",
"age": 10,
"color": "white"
}
]
},
{
"dogs": [
{
"name": "sam",
"age": 5,
"color": "black and white"
},
{
"name": "rover",
"age": 2,
"color": "brown and white"
}
]
},
{
"snakes": [
{
"name": "noodles",
"age": 10,
"color": "green"
}
]
}
],
"owners": [
"peter",
"william",
"sally"
]
}
Suppose you had an (a priori) indeterminate or large numbers of types of animals, and just one owners file. In such cases, it would be better (to save memory) not to use the -s option, and it would be easier to invoke jq with the owners file as the first data file, e.g. along the lines of:
jq -n -f program.jq owners.json $(ls *.json | grep -v owners.json)
where program.jq contains a program such as:
input as $owners | {$owners, animals: [inputs]}
(Notice how {"owners": $owners} can be abbreviated.)
Not sure if this is the way-to-go, but it gets the desired output by:
Using --slurp:
Catching the first 3 files as a single array variable
[ .[0] * .[1] * .[2] ] as $all
Catching owners object as a single variable
.[3].owners as $owners
Creating the object as desired
{ "animals": $all, "owners": $owners }
jq \
--slurp \
'[ .[0] * .[1] * .[2] ] as $all | .[3].owners as $owners | { "animals": $all, "owners": $owners }' cats.json dogs.json snakes.json owners.json
Will produce:
{
"animals": [
{
"cats": [
{
"name": "fluffles",
"age": 10,
"color": "white"
}
],
"dogs": [
{
"name": "sam",
"age": 5,
"color": "black and white"
},
{
"name": "rover",
"age": 2,
"color": "brown and white"
}
],
"snakes": [
{
"name": "noodles",
"age": 10,
"color": "green"
}
]
}
],
"owners": [
"peter",
"william",
"sally"
]
}

JQ convert json array to object properties

I have a json structured like this:
[
{
"name": "object1",
"prop": "prop1",
"props": [
{ "prop1": "value1" },
{ "prop2": "value2" },
{ "prop3": "value3" }
]
},
{
"name": "object2",
"prop": "prop2",
"props": [
{ "prop1": "value4" },
{ "prop2": "value5" },
{ "prop3": "value6" }
]
}
]
I would like to extract the content of the props variable and have it as extra object properties, looking like this:
[
{
"name": "object1",
"prop": "prop1",
"prop1": "value1",
"prop2": "value2",
"prop3": "value3"
},
{
"name": "object2",
"prop": "prop2",
"prop1": "value4",
"prop2": "value5",
"prop3": "value6"
},
]
I've been trying to use map but I can't seem to be able to get rid of the array.
Use add to merge the propN objects into one, del to remove the original props:
jq '[.[] | . + (.props | add) | del(.props)]' file.json
You can indeed use map to shorten it a bit:
jq 'map(. + (.props | add) | del(.props))' file.json

How to select/get object elements' (array of objects) keys based on value of an element in an object in the aforementioned array in jq?

I have the following input:
{
"key1": {
"subkey1": [
{
"filterkey1": "value1",
"filterkey2": "value2"
},
{
"filterkey1": "value3",
"filterkey2": "value4"
}
],
"subkey2": [
{
"filterkey1": "value5",
"filterkey2": "value6"
},
{
"filterkey1": "value7",
"filterkey2": "value8"
}
],
"subkey3": [
{
"filterkey1": "value1",
"filterkey2": "value6"
},
{
"filterkey1": "value9",
"filterkey2": "value4"
}
]
},
"key2": {
}
}
I want to get the key of the arrays that has an object which has "value1" for key: "filterkey1". So in this case the output must be:
["subkey1", "subkey3"]
All the elements I care about are in "key1" object.
Get subkeys using keys_unsorted, and check if their values have filterkey1: "value1" pair using any:
.key1 | [
keys_unsorted[] as $k
| if any(.[$k][]; .filterkey1=="value1")
then $k
else empty end
]