I have a json file like this
[
"public-policy-routing/0",
"public-policy-routing/1",
"public-policy-routing/5",
"public-policy-routing/7",
"public-policy-routing/10"
]
since the element has pattern "key/values" where only values part differ I want to reduce the output to something like this
{ "public-policy-routing" : [0,1,5,7,10] }
any idea how to achieve above using jq ?
One way would be using reduce on the array elements. Split at the slash, destructure the resulting array into a key and value variable, then add the value as an array to the field with the key name:
jq 'reduce (.[]/"/") as [$k,$v] ({}; .[$k] += [$v])'
{
"public-policy-routing": [
"0",
"1",
"5",
"7",
"10"
]
}
Demo
If you wanted numbers instead of strings, convert the values using tonumber:
jq 'reduce (.[]/"/") as [$k,$v] ({}; .[$k] += [$v | tonumber])'
{
"public-policy-routing": [
0,
1,
5,
7,
10
]
}
Demo
Here is a mildly robust solution:
def ton: try tonumber catch .;
def keysplit: index("/") as $i
| if $i then [.[:$i], (.[$i+1:]|ton) ] else [., ""] end;
reduce ( .[]|keysplit ) as [$k,$v] ({}; .[$k] += [$v] )
Here is a robust solution that works with arbitrary strings by grouping them:
def group(f): group_by(f) | map({key:first|f, value:.}) | from_entries;
map(. / "/") | group(first) | map_values(map(last|tonumber))
or without a custom function:
map(. / "/")
| group_by(first)
| map({key:first|first, value:map(last|tonumber)})
| from_entries
Related
Given an arbitrary JSON input:
{
"id":"038020",
"title":"Teenage Mutant Ninja Turtles: Out of the Shadows",
"turtles":[
{
"name":"Leonardo",
"mask":"blue"
},
{
"name":"Michelangelo",
"mask":"orange"
},
{
"name":"Donatello",
"mask":"purple"
},
{
"name":"Raphael",
"mask":"red"
}
],
"summary":"The Turtles continue to live in the shadows and no one knows they were the ones who took down Shredder",
"cast":"Megan Fox, Will Arnett, Tyler Perry",
"director":"Dave Green"
}
And an arbitrary list of JQ paths like [".turtles[].name", ".cast", ".does.not.exist"], or any similar format
How can I create new JSON with only the information contained in the paths of the list?
In this case the expected result would be:
{
"turtles":[
{
"name":"Leonardo"
},
{
"name":"Michelangelo"
},
{
"name":"Donatello"
},
{
"name":"Raphael"
}
],
"cast":"Megan Fox, Will Arnett, Tyler Perry"
}
I've seen similar solutions in problems like "removing null entries" from a JSON using the walk function present in jq1.5+, somewhat along the lines of:
def filter_list(input, list):
input
| walk(
if type == "object" then
with_entries( select(.key | IN( list )))
else
.
end);
filter_list([.], [.a, .b, .c[].d])
But it should take in account the full path in the JSON somehow.
What is the best approach to solve this problem?
If $paths contains an array of explicit jq paths (such as [ ["turtles", 0, "name"], ["cast"]]), the simplest approach would be to
use the following filter:
. as $in
| reduce $paths[] as $p (null; setpath($p; $in | getpath($p)))
Extended path expressions
In order to be able to handle extended path expressions such as ["turtles", [], "name"], where [] is intended to range over the indices of the turtles array, we shall define the following helper function:
def xpath($ary):
. as $in
| if ($ary|length) == 0 then null
else $ary[0] as $k
| if $k == []
then range(0;length) as $i | $in[$i] | xpath($ary[1:]) | [$i] + .
else .[$k] | xpath($ary[1:]) | [$k] + .
end
end ;
For the sake of exposition, let us also define:
def paths($ary): $ary[] as $path | xpath($path);
Then with the given input, the expression:
. as $in
| reduce paths([ ["turtles", [], "name"], ["cast"]]) as $p
(null; setpath($p; $in | getpath($p)) )
produces the output shown below.
Using path
It is worth point out that one way to handle expressions such as ".turtles[].name" would be to use the builtin filter path/1.
For example:
# Emit a stream of paths:
def paths: path(.turtles[].name), ["cast"];
. as $in
| reduce paths as $p (null; setpath($p; $in | getpath($p)))
Output:
{
"turtles": [
{
"name": "Leonardo"
},
{
"name": "Michelangelo"
},
{
"name": "Donatello"
},
{
"name": "Raphael"
}
],
"cast": "Megan Fox, Will Arnett, Tyler Perry"
}
I'm trying to use jq to parse Solr 6.5 metrics into key value pairs:
{
"responseHeader": {
"status": 0,
"QTime": 7962
},
"metrics": [
"solr.core.shard1",
"QUERY./select",
"solr.core.shard2",
"QUERY./update"
...
]
}
I'd like to pick even odd entries in metrics array and put them together into a single object as key value pairs like this:
{
"solr.core.shard1": "QUERY./select",
"solr.core.shard2": "QUERY./update",
...
}
Till now, I am only able to come up with:
.metrics | to_entries | .[] | {(select(.key % 2 == 0).value): select(.key % 2 == 1).value}
But this returns an error or no results.
I'd be grateful if someone could point me in the right direction. I feel like the answer is probably in the map operator, but I haven't been able to figure it out.
jq solution:
jq '[ .metrics as $m | range(0; $m | length; 2)
| {($m[.]): $m[(. + 1)]} ] | add' jsonfile
The output:
{
"solr.core.shard1": "QUERY./select",
"solr.core.shard2": "QUERY./update"
}
https://stedolan.github.io/jq/manual/v1.5/#range(upto),range(from;upto)range(from;upto;by)
Here's a helper function which makes the solution trivial:
# Emit a stream consisting of pairs of items taken from `stream`
def pairwise(stream):
foreach stream as $i ([];
if length == 1 then . + [$i] else [$i] end;
select(length == 2));
From here there are several good options, e.g. we could start with:
.metrics
| [pairwise(.[]) | {(.[0]): .[1]}]
| add
With your input, this produces:
{
"solr.core.shard1": "QUERY./select",
"solr.core.shard2": "QUERY./update"
}
So you might want to write:
.metrics |= ([pairwise(.[]) | {(.[0]): .[1]}] | add)
I am trying to sort by key in an arbitrary format (key 'event' should be the first)
I know this one sort the key by alphabetical order:
jq -S '.' file.json
but is there a function to sort key so that the first one is always the same and not by alphabetical order?
It is to make them more human readable and have the most significant key first
Currently have:
{key1:value, shouldbeFirstKey:value2, ...}
Would like
{shouldbeFirstKey:value2, key1:value, ...}
Suppose we have an object with a bunch of keys, and want some of those keys to appear in a certain order, while leaving the others as-is. Then the technique illustrated by the following example can be used:
$ jq -n '{a:2,b:3,first:0,second:1} | . as $in | {first,second} + $in'
The result:
{
"first": 0,
"second": 1,
"a": 2,
"b": 3
}
rekey
Let's call the object defining the key ordering as the "template object" ({first,second} above). Notice that using the technique described above, the keys in the "template object" always appear in the result. If we only want the template object keys to appear in the result if they appear in the input, we can modify the above approach using the following function:
def rekey(obj):
. as $in
| reduce (obj|keys_unsorted)[] as $k ({};
if $in|has($k) then . + {($k): $in[$k]} else . end)
| . + $in ;
For example:
{a:2,b:3,first:0,second:1} | rekey({first,second,third})
produces:
{
"first": 0,
"second": 1,
"a": 2,
"b": 3
}
With walk/1
If one wants to reorder keys recursively, one can use walk/1 (defined as at https://github.com/stedolan/jq/blob/master/src/builtin.jq),
as illustrated here using the above definition of rekey:
walk(if type == "object" then rekey($template) else . end)
where $template represents the "template object".
There is no need to define a special variant of walk/1. Simply define a function that takes as input an arbitrary object, and that produces the desired reordering.
(If your jq comes with the version of walk/1 that uses keys, then you should consider updating your jq, or redefine walk/1 to use keys_unsorted.)
Here is a filter (derived from walk/1 [source]) which will reorder the keys of subobjects using a function:
def reorder(order):
. as $in
| if type == "object" then
reduce (keys_unsorted|order)[] as $key(
{}; . + { ($key): ($in[$key] | reorder(order)) }
)
elif type == "array" then map( reorder(order) )
else .
end;
the order function is expected to return the set of keys in whatever order is desired. e.g. the following function moves "shouldbeFirstKey" to the first position
def neworder:
"shouldbeFirstKey" as $s
| index($s) as $i
| if $i==null then . else [$s] + .[:$i] + .[$i+1:] end
;
so that
{key1:"value", shouldbeFirstKey:"value2", other:{key3:"value3", shouldbeFirstKey:"value4"}}
| reorder(neworder)
produces the output
{
"shouldbeFirstKey": "value2",
"key1": "value",
"other": {
"shouldbeFirstKey": "value4",
"key3": "value3"
}
}
Try it online!
idea from #jq170727 's answer:
jq '
def reorder(a; n):
def order(a; n):
a as $a |
n as $n |
index($a) as $i |
if $i == null then .
else
if $i<$n then .[:$i] + .[$i+1:$n+1] + [$a] + .[$n+1:]
elif $n<$i then .[:$n] + [$a] + .[$n:]
else .
end
end;
. as $in
| if type == "object" then
reduce (keys_unsorted|order(a;n))[] as $key(
{}; . + { ($key): ($in[$key] | reorder(a; n)) }
)
elif type == "array" then map( reorder(a; n) )
else .
end;
[.[] | reorder("id"; 0)| reorder("size"; 3) | reorder("name"; 1)]'
data.json
{
"prodid_876006": {
"serid": [{
"seridone": "3265874"
}, {
"seridtwo": "21458915"
}],
"serials": ["028915"]
},
"prodid_980": {
"serid": [{
"seridone": "32743214"
}, {
"seridtwo": "5469872"
}],
"serials": ["192147","1632589"]
}
}
% jq '[to_entries[] | {"key": .key, "value": .value.serials}] | from_entries' some.json
gives:
{
"prodid_876006": [
"028915"
],
"prodid_980": [
"192147",
"1632589"
]
}
how could i get the following output? (here the key is each element of the serials array, and the value is the key of the sample json):
{
"028915" : ["prodid_876006"],
"192147" : ["prodid_980"],
"1632589" : ["prodid_980"]
}
The following solution uses reduce within reduce and so might look a bit tricky, so it might help to think of "reduce" as a kind of "do loop", as suggested by the indentation:
reduce to_entries[] as $kv ({};
reduce $kv.value.serials[] as $s (.;
. + {($s): (.[$s] + [$kv|.key]) } ) )
Here is a solution which uses to_entries, reduce and add
[ to_entries[]
| .key as $k
| reduce .value.serials[] as $v ({}; .[$v] += [$k])
] | add
To start, a more succinct way to get the first result in your question is to use map_values. You're changing all the values in your object map to the contents of the serials value. Which can be expressed simply as:
map_values(.serials)
From there, it's just a matter of pivoting the data.
map_values(.serials) | reduce to_entries[] as $e ({};
.[$e.value[]] += [$e.key]
)
I have following json:
{
"vertices": [
{
"__cp": "foo",
"__type": "metric",
"__eid": "foobar",
"name": "Undertow Metrics~Sessions Created",
"_id": 45056,
"_type": "vertex"
},
...
]
"edges": [
...
and I would like to achieve this format:
{
"nodes": [
{
"cp": "foo",
"type": "metric",
"label": "metric: Undertow Metrics~Sessions Created",
"name": "Undertow Metrics~Sessions Created",
"id": 45056
},
...
]
"edges": [
...
So far I was able to create this expression:
jq '{nodes: .vertices} | del(.nodes[]."_type", .nodes[]."__eid")'
I.e. rename 'vertices' to 'nodes' and remove '_type' and '__eid', how can I rename a key nested deeper in the JSON?
You can change the names of properties of objects if you use with_entries(filter). This converts an object to an array of key/value pairs and applies a filter to the pairs and converts back to an object. So you would just want to update the key of those objects to your new names.
Depending on which version of jq you're using, the next part can be tricky. String replacement doesn't get introduced until jq 1.5. If that was available, you could then do this:
{
nodes: .vertices | map(with_entries(
.key |= sub("^_+"; "")
)),
edges
}
Otherwise if you're using jq 1.4, then you'll have to remove them manually. A recursive function can help with that since the number of underscores varies.
def ltrimall(str): str as $str |
if startswith($str)
then ltrimstr($str) | ltrimall(str)
else .
end;
{
nodes: .vertices | map(with_entries(
.key |= ltrimall("_")
)),
edges
}
The following program works with jq 1.4 or jq 1.5.
It uses walk/1 to remove leading underscores from any key, no matter where it occurs in the input JSON.
The version of ltrim provided here uses recurse/1 for efficiency and portability, but any suitable substitute may be used.
def ltrim(c):
reduce recurse( if .[0:1] == c then .[1:] else null end) as $x
(null; $x);
# Apply f to composite entities recursively, and to atoms
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
.vertices = .nodes
| del(.nodes)
| (.vertices |= walk(
if type == "object"
then with_entries( .key |= ltrim("_") )
else .
end ))
From your example data it looks like you intend lots of little manipulations so I'd break things out into stages like this:
.nodes = .vertices # \ first take care of renaming
| del(.vertices) # / .vertices to .nodes
| .nodes = [
.nodes[] # \ then scan each node
| . as $n # /
| del(._type, .__eid) # \ whatever key-specific tweaks like
| .label = "metric: \(.name)" # / calculating .label you want can go here
| reduce keys[] as $k ( # \
{}; # | final reduce to handle renaming
.[$k | sub("^_+";"")] = $n[$k] # | any keys that start with _
) # /
]