{
"prodid_876006": {
"serid": [{
"seridone": "3265874"
}, {
"seridtwo": "21458915"
}],
"serials": ["028915"]
},
"prodid_980": {
"serid": [{
"seridone": "32743214"
}, {
"seridtwo": "5469872"
}],
"serials": ["192147","1632589"]
}
}
% jq '[to_entries[] | {"key": .key, "value": .value.serials}] | from_entries' some.json
gives:
{
"prodid_876006": [
"028915"
],
"prodid_980": [
"192147",
"1632589"
]
}
how could i get the following output? (here the key is each element of the serials array, and the value is the key of the sample json):
{
"028915" : ["prodid_876006"],
"192147" : ["prodid_980"],
"1632589" : ["prodid_980"]
}
The following solution uses reduce within reduce and so might look a bit tricky, so it might help to think of "reduce" as a kind of "do loop", as suggested by the indentation:
reduce to_entries[] as $kv ({};
reduce $kv.value.serials[] as $s (.;
. + {($s): (.[$s] + [$kv|.key]) } ) )
Here is a solution which uses to_entries, reduce and add
[ to_entries[]
| .key as $k
| reduce .value.serials[] as $v ({}; .[$v] += [$k])
] | add
To start, a more succinct way to get the first result in your question is to use map_values. You're changing all the values in your object map to the contents of the serials value. Which can be expressed simply as:
map_values(.serials)
From there, it's just a matter of pivoting the data.
map_values(.serials) | reduce to_entries[] as $e ({};
.[$e.value[]] += [$e.key]
)
Related
I have a json file like this
[
"public-policy-routing/0",
"public-policy-routing/1",
"public-policy-routing/5",
"public-policy-routing/7",
"public-policy-routing/10"
]
since the element has pattern "key/values" where only values part differ I want to reduce the output to something like this
{ "public-policy-routing" : [0,1,5,7,10] }
any idea how to achieve above using jq ?
One way would be using reduce on the array elements. Split at the slash, destructure the resulting array into a key and value variable, then add the value as an array to the field with the key name:
jq 'reduce (.[]/"/") as [$k,$v] ({}; .[$k] += [$v])'
{
"public-policy-routing": [
"0",
"1",
"5",
"7",
"10"
]
}
Demo
If you wanted numbers instead of strings, convert the values using tonumber:
jq 'reduce (.[]/"/") as [$k,$v] ({}; .[$k] += [$v | tonumber])'
{
"public-policy-routing": [
0,
1,
5,
7,
10
]
}
Demo
Here is a mildly robust solution:
def ton: try tonumber catch .;
def keysplit: index("/") as $i
| if $i then [.[:$i], (.[$i+1:]|ton) ] else [., ""] end;
reduce ( .[]|keysplit ) as [$k,$v] ({}; .[$k] += [$v] )
Here is a robust solution that works with arbitrary strings by grouping them:
def group(f): group_by(f) | map({key:first|f, value:.}) | from_entries;
map(. / "/") | group(first) | map_values(map(last|tonumber))
or without a custom function:
map(. / "/")
| group_by(first)
| map({key:first|first, value:map(last|tonumber)})
| from_entries
I would like to "transpose" (not sure that's the right word) JSON elements.
For example, I have a JSON file like this:
{
"name": {
"0": "fred",
"1": "barney"
},
"loudness": {
"0": "extreme",
"1": "not so loud"
}
}
... and I would like to generate a JSON array like this:
[
{
"name": "fred",
"loudness": "extreme"
},
{
"name": "barney",
"loudness": "not so loud"
}
]
My original JSON has many more first level elements than just "name" and "loudness", and many more names, features, etc.
For this simple example I could fully specify the transformation like this:
$ echo '{"name":{"0":"fred","1":"barney"},"loudness":{"0":"extreme","1":"not so loud"}}'| \
> jq '[{"name":.name."0", "loudness":.loudness."0"},{"name":.name."1", "loudness":.loudness."1"}]'
[
{
"name": "fred",
"loudness": "extreme"
},
{
"name": "barney",
"loudness": "not so loud"
}
]
... but this isn't feasible for the original JSON.
How can jq create the desired output while being key-agnostic for my much larger JSON file?
Yes, transpose is an appropriate word, as the following makes explicit.
The following generic helper function makes for a simple solution that is completely agnostic about the key names, both of the enclosing object and the inner objects:
# Input: an array of values
def objectify($keys):
. as $in | reduce range(0;length) as $i ({}; .[$keys[$i]] = $in[$i]);
Assuming consistency of the ordering of the inner keys
Assuming the key names in the inner objects are given in a consistent order, a solution can now obtained as follows:
keys_unsorted as $keys
| [.[] | [.[]]] | transpose
| map(objectify($keys))
Without assuming consistency of the ordering of the inner keys
If the ordering of the inner keys cannot be assumed to be consistent, then one approach would be to order them, e.g. using this generic helper function:
def reorder($keys):
. as $in | reduce $keys[] as $k ({}; .[$k] = $in[$k]);
or if you prefer a reduce-free def:
def reorder($keys): [$keys[] as $k | {($k): .[$k]}] | add;
The "main" program above can then be modified as follows:
keys_unsorted as $keys
| (.[$keys[0]]|keys_unsorted) as $inner
| map_values(reorder($inner))
| [.[] | [.[]]] | transpose
| map(objectify($keys))
Caveat
The preceding solution only considers the key names in the first inner object.
Building upon Peak's solution, here is an alternative based on group_by to deal with arbitrary orders of inner keys.
keys_unsorted as $keys
| map(to_entries[])
| group_by(.key)
| map(with_entries(.key = $keys[.key] | .value |= .value))
Using paths is a good idea as pointed out by Hobbs. You could also do something like this :
[ path(.[][]) as $p | { key: $p[0], value: getpath($p), id: $p[1] } ]
| group_by(.id)
| map(from_entries)
This is a bit hairy, but it works:
. as $data |
reduce paths(scalars) as $p (
[];
setpath(
[ $p[1] | tonumber, $p[0] ];
( $data | getpath($p) )
)
)
First, capture the top level as $data because . is about to get a new value in the reduce block.
Then, call paths(scalars) which gives a key path to all of the leaf nodes in the input. e.g. for your sample it would give ["name", "0"] then ["name", "1"], then ["loudness", "0"], then ["loudness", "1"].
Run a reduce on each of those paths, starting the reduction with an empty array.
For each path, construct a new path, in the opposite order, with numbers-in-strings turned into real numbers that can be used as array indices, e.g. ["name", "0"] becomes [0, "name"].
Then use getpath to get the value at the old path in $data and setpath to set a value at the new path in . and return it as the next . for the reduce.
At the end, the result will be
[
{
"name": "fred",
"loudness": "extreme"
},
{
"name": "barney",
"loudness": "not so loud"
}
]
If your real data structure might be two levels deep then you would need to replace [ $p[1] | tonumber, $p[0] ] with a more appropriate expression to transform the path. Or maybe some of your "values" are objects/arrays that you want to leave alone, in which case you probably need to replace paths(scalars) with something like paths | select(length == 2).
I have a complex JSON file that contains hundreds of "attributes" with their types identified by "objectTypeAttributeId".
I know that objectTypeAttributeId=328 means tickedid, objectTypeAttributeId=329 contains array of hostnames etc..
There is simplified version of the file:
{
"objectEntries": [
{
"attributes": [
{
"id": 279792,
"objectTypeAttributeId": 328,
"objectAttributeValues": [
{
"displayValue": "ITSM-24210"
}
]
},
{
"id": 279795,
"objectTypeAttributeId": 329,
"objectAttributeValues": [
{
"displayValue": "testhost1"
},
{
"displayValue": "testhost2"
}
]
},
{
"id": 279793,
"objectTypeAttributeId": 330,
"objectAttributeValues": [
{
"displayValue": "28.02.2020 11:45"
}
]
}
]
}
]
}
I need to create output JSON using particular values picked out (according to the "objectTypeAttributeId" value) of input JSON in format like this:
{
"tickets": [
{
"ticketid": "ITSM-24210",
"hostnames": ["testhost1", "testhost2"],
"date": "28.02.2020 11:45"
}
]
}
I am new in jq, in the XSLT it is solvable using static template with placeholders for picked values.
I have tried this approach, there is my jq filter:
.objectEntries[].attributes[] |
{ticketid: select(.objectTypeAttributeId == 328) | .objectAttributeValues[0].displayValue},
{hostnames: select(.objectTypeAttributeId == 329) | [.objectAttributeValues[].displayValue]},
{date: select(.objectTypeAttributeId == 330) | .objectAttributeValues[0].displayValue}
but the result of this approach is:
{
"ticketid": "ITSM-24210"
}
{
"hostnames": [
"testhost1",
"testhost2"
]
}
{
"date": "28.02.2020 11:45"
}
And all my subsequent tries to format output better ends in broken jq filter or filter that does not return anything.
Please any ideas how to solve this problem?
Assuming a ticket is to be generated for each object entry:
{tickets: [
.objectEntries[]
| [.attributes[]
| [.objectTypeAttributeId,
(.objectAttributeValues | map(.displayValue))] as [$id, $val]
| if $id == 328 then {ticketId: $val[0]}
elif $id == 329 then {hostnames: $val}
elif $id == 330 then {date: $val[0]}
else empty end
] | add
]}
Online demo
Here we go, it's not pretty, there may be a better solution but it works: https://jqplay.org/s/sxussfa2Vj
.objectEntries | {tickets: map(.attributes |
{ticketID: (reduce .[] as $r (null; if $r.objectTypeAttributeId == 328
then $r.objectAttributeValues[0].value else . end)),
date: (reduce .[] as $r (null; if $r.objectTypeAttributeId == 330
then $r.objectAttributeValues[0].value else . end)),
hostnames: (reduce .[] as $r ([]; if $r.objectTypeAttributeId == 329
then $r.objectAttributeValues | map(.value) else . end))})}
There's a lot of unpacking and repacking going on here that sort of distracts from the core. You have an array of tickets (aka entries), and over those we map. The various properties we have to grab from different entries of an array, which is done using reduce. Reduce goes through the array of objects and picks out the right one and keeps track of the value.
Maybe there's a nice way, but this works already, so you can play with it further, trying to simplify.
Your original solution almost works, you did a good job there, just needed a map:
.objectEntries[].attributes |
{ticketid: . | map(select(.objectTypeAttributeId == 328))[0] |
.objectAttributeValues[0].displayValue,
date: . | map(select(.objectTypeAttributeId == 330))[0] |
.objectAttributeValues[0].displayValue,
hostnames: . | map(select(.objectTypeAttributeId == 329))[0] |
[.objectAttributeValues[].displayValue]}
Try it out, it even works with multiple tickets ;)
https://jqplay.org/s/ydoCgv9vsI
Given an arbitrary JSON input:
{
"id":"038020",
"title":"Teenage Mutant Ninja Turtles: Out of the Shadows",
"turtles":[
{
"name":"Leonardo",
"mask":"blue"
},
{
"name":"Michelangelo",
"mask":"orange"
},
{
"name":"Donatello",
"mask":"purple"
},
{
"name":"Raphael",
"mask":"red"
}
],
"summary":"The Turtles continue to live in the shadows and no one knows they were the ones who took down Shredder",
"cast":"Megan Fox, Will Arnett, Tyler Perry",
"director":"Dave Green"
}
And an arbitrary list of JQ paths like [".turtles[].name", ".cast", ".does.not.exist"], or any similar format
How can I create new JSON with only the information contained in the paths of the list?
In this case the expected result would be:
{
"turtles":[
{
"name":"Leonardo"
},
{
"name":"Michelangelo"
},
{
"name":"Donatello"
},
{
"name":"Raphael"
}
],
"cast":"Megan Fox, Will Arnett, Tyler Perry"
}
I've seen similar solutions in problems like "removing null entries" from a JSON using the walk function present in jq1.5+, somewhat along the lines of:
def filter_list(input, list):
input
| walk(
if type == "object" then
with_entries( select(.key | IN( list )))
else
.
end);
filter_list([.], [.a, .b, .c[].d])
But it should take in account the full path in the JSON somehow.
What is the best approach to solve this problem?
If $paths contains an array of explicit jq paths (such as [ ["turtles", 0, "name"], ["cast"]]), the simplest approach would be to
use the following filter:
. as $in
| reduce $paths[] as $p (null; setpath($p; $in | getpath($p)))
Extended path expressions
In order to be able to handle extended path expressions such as ["turtles", [], "name"], where [] is intended to range over the indices of the turtles array, we shall define the following helper function:
def xpath($ary):
. as $in
| if ($ary|length) == 0 then null
else $ary[0] as $k
| if $k == []
then range(0;length) as $i | $in[$i] | xpath($ary[1:]) | [$i] + .
else .[$k] | xpath($ary[1:]) | [$k] + .
end
end ;
For the sake of exposition, let us also define:
def paths($ary): $ary[] as $path | xpath($path);
Then with the given input, the expression:
. as $in
| reduce paths([ ["turtles", [], "name"], ["cast"]]) as $p
(null; setpath($p; $in | getpath($p)) )
produces the output shown below.
Using path
It is worth point out that one way to handle expressions such as ".turtles[].name" would be to use the builtin filter path/1.
For example:
# Emit a stream of paths:
def paths: path(.turtles[].name), ["cast"];
. as $in
| reduce paths as $p (null; setpath($p; $in | getpath($p)))
Output:
{
"turtles": [
{
"name": "Leonardo"
},
{
"name": "Michelangelo"
},
{
"name": "Donatello"
},
{
"name": "Raphael"
}
],
"cast": "Megan Fox, Will Arnett, Tyler Perry"
}
Using jq I'd like to convert data of the format:
{
"key": "something-else",
"value": {
"value": "bloop",
"isEncrypted": false
}
}
{
"key": "something",
"value": {
"value": "blah",
"isEncrypted": false
}
}
To the format:
{
something: "blah",
something-else: "bloop"
}
Filtering out 'encrypted values' along the way. How can I achieve this? I've gotten as far as the following:
.parameters | to_entries[] | select (.value.isEncrypted == false) | .key + ": " + .value.value
Which produces:
"something-else: bloop"
"something: blah"
Close, but not there just yet. I suspect that there's some clever function for this.
Given the example input, here's a simple solution, assuming the stream of objects is available as an array. (This can be done using jq -s if the JSON objects are given as input to jq, or in your case, following your example, simply using .parameters | to_entries).
map( select(.value.isEncrypted == false) | {(.key): .value.value } )
| add
This produces the JSON object:
{
"something-else": "bloop",
"something": "blah"
}
The key ideas here are:
the syntax for object construction: {( KEYNAME ): VALUE}
add
One way to gain an understanding of how this works is to run the first part of the filter (map(...)) first.
Using keys_unsorted
If you want to avoid the overhead of to_entries, you might want to consider the following approach, which piggy-backs off your implicit description of .parameters:
.parameters
| [ keys_unsorted[] as $k
| if .[$k].isEncrypted == false
then { ($k) : .[$k].value } else empty end ]
| add