Using jq, how can I transform:
{ "a": {"b": 0}, "c": {"d": 1}}
into:
{"b": {"a": 0}, "d": {"c": 1}}
without knowing the name of the keys in the source?
(I know that this can lose data in the general case, but not with my data)
Here's an alternative using with_entries:
with_entries(.key as $parent
| (.value|keys[0]) as $child
| {
key: $child,
value: { ($parent): .value[$child] }
}
)
def swapper:
. as $in
| reduce keys[] as $key
( {}; . + ( $in[$key] as $o
| ($o|keys[0]) as $innerkey
| { ($innerkey): { ($key): $o[$innerkey] } } ) ) ;
Example:
{ "a": {"b": 0}, "c": {"d": 1}} | swapper
produces:
{"b":{"a":0},"d":{"c":1}}
Here is a solution which uses jq streams and variables:
[
. as $d
| keys[]
| $d[.] as $v
| ($v|keys[]) as $vkeys
| {
($vkeys): {
(.): ($vkeys|$v[.])
}
}
] | add
It is easy to lose track of what is what at the end so to see more clearly what's going on here is a slightly expanded version with additional comments and variables.
[
. as $d # $d: {"a":{"b":0},"c":{"d": 1}}
| keys[] | . as $k # $k: "a", "c"
| $d[$k] as $v # $v: {"b": 0}, {"d": 1}
| ($v|keys[]) as $vkeys # $vkeys: "b", "d"
| ($vkeys|$v[.]) as $vv # $vv: 0, 1
| {
($vkeys): { # "b": { "d": {
($k): $vv # "a": 0 "c": 1
} # } , }
}
] | add
Related
Using jq we can easily merge two multi-level objects X and Y using *:
X='{
"a": 1,
"b": 5,
"c": {
"a": 3
}
}' Y='{
"d": 2,
"a": 3,
"c": {
"x": 10,
"y": 11
}
}' && Z=`echo "[$X,$Y]"|jq '.[0] * .[1]'` && echo "Z='$Z'"
gives us:
Z='{
"a": 3,
"b": 5,
"c": {
"a": 3,
"x": 10,
"y": 11
},
"d": 2
}'
But in my case, I'm starting with X and Z and want to calculate Y (such that X * Y = Z). If we only have objects with scalar properties, then jq X + Y equals Z, and we can also calculate Y as jq Z - X. However, this fails if X or Y contain properties with object values such as in the above example:
X='{
"a": 1,
"b": 5,
"c": {
"a": 3
}
}' Z='{
"a": 3,
"b": 5,
"c": {
"a": 3,
"x": 10,
"y": 11
},
"d": 2
}' && echo "[$X,$Z]" | jq '.[1] - .[0]'
throws an error jq: error (at <stdin>:16): object ({"a":3,"b":...) and object ({"a":1,"b":...) cannot be subtracted
Is there an elegant solution to this problem with jq?
UPDATE: I've accepted the answer that I found easier to read / maintain and with superior performance. In addition, I found a wrinkle in my need which was that if X contained a key K that was not present in Z, I needed the output (Y) to nullify it by containing the key K with a value of null.
The best way I could come up with to do this was to pre-process Z to add the missing keys using the below:
def add_null($y):
reduce (to_entries[] | [ .key, .value ] ) as [ $k, $v ] (
$y;
if $y | has($k) | not then
.[$k] = null
elif $v | type == "object" then
.[$k] = ($v | add_null($y[$k]))
else
.[$k] = $v
end
);
so we end up with:
def add_null(...);
def remove(...);
. as [ $X, $Z ] | ($X | add_null($Z)) | remove($X)
Any better suggestions to this variation are still appreciated!
def remove($o2):
reduce ( to_entries[] | [ .key, .value ] ) as [ $k, $v1 ] (
{};
if $o2 | has($k) | not then
# Keep existing value if $o2 doesn't have the key.
.[$k] = $v1
else
$o2[$k] as $v2 |
if $v1 | type == "object" then
# We're comparing objects.
( $v1 | remove($o2[$k]) ) as $v_diff |
if $v_diff | length == 0 then
# Discard identical values.
.
else
# Keep the differences of the values.
.[$k] = $v_diff
end
else
# We're comparing non-objects.
if $v1 == $v2 then
# Discard identical values.
.
else
# Keep existing value if different.
.[$k] = $v1
end
end
end
);
. as [ $Z, $X ] | $Z | remove($X)
Demo on jqplay
or
def sub($v2):
( type ) as $t1 |
( $v2 | type ) as $t2 |
if $t1 == $t2 then
if $t1 == "object" then
with_entries(
.key as $k |
.value = (
.value |
if $v2 | has($k) then sub( $v2[$k] ) else . end
)
) |
select( length != 0 )
else
select( . != $v2 )
end
else
.
end;
. as [ $Z, $X ] | $Z | sub($X)
Demo on jqplay
I don't know if this is elegant, but it works for your sample data
echo "[$X,$Z]" | jq '
. as [$x,$z]
| map([paths(scalars)])
| .[0] |= map(select(. as $p | [$x, $z | getpath($p)] | .[1] == .[0]))
| reduce (.[1] - .[0])[] as $p (null; setpath($p; $z | getpath($p)))
'
{
"a": 3,
"c": {
"x": 10,
"y": 11
},
"d": 2
}
Demo
I have an array of objects, and want to filter the arrays in the b property to only have elements matching the a property of the object.
[
{
"a": 3,
"b": [
1,
2,
3
]
},
{
"a": 5,
"b": [
3,
5,
4,
3,
5
]
}
]
produces
[
{
"a": 3,
"b": [
3
]
},
{
"a": 5,
"b": [
5,
5
]
}
]
Currently, I've arrived at
[.[] | (.a as $a | .b |= [.[] | select(. == $a)])]
That works, but I'm wondering if there's a better (shorter, more readable) way.
I can think of two ways to do this with less code and both are variants of what you have already figured out on your own.
map(.a as $a | .b |= map(select(. == $a)))
del(.[] | .a as $a | .b[] | select(. != $a))
I have a JSON test.json as follows:
[
{
"a": "a",
"b": [
"a",
"b",
"c"
]
},
{
"a": "a"
}
]
And I would want to join the field b of each entry and handle the case of its
emptiness:
{ "a": "a",
"b": "a, b, c"
},
{
"a": "a",
"b": null
}
The following command works...
cat test.json |
jq '.[] | .b as $t | if $t then {a: .a, b: $t | join(", ")} else {a: .a, b: $t} end'
... but it's too long as I have to write almost the same constructor two times.
I have tried to move the if-then-else conditional or even the // operator in the {} construction, but they result to a syntax error.
Depending on how you want to handle null/empty values you could try these:
map(.b |= (. // [] | join(", ")))
map(if .b then .b |= join(", ") else . end)
I've been trying to convert some JSON to csv and I have the following problem:
I have the following input json:
{"id": 100, "a": [{"t" : 1,"c" : 2 }, {"t": 2, "c" : 3 }] }
{"id": 200, "a": [{"t": 2, "c" : 3 }] }
{"id": 300, "a": [{"t": 1, "c" : 3 }] }
And I expect the following CSV output:
id,t1,t2
100,2,3
200,,3
300,3,
Unfortunately JQ doesn't output if one of select has no match.
Example:
echo '{ "id": 100, "a": [{"t" : 1,"c" : 2 }, {"t": 2, "c" : 3 }] }' | jq '{t1: (.a[] | select(.t==1)).c , t2: (.a[] | select(.t==2)).c }'
output:
{ "t1": 2, "t2": 3 }
but if one of the objects select returns no match it doesn't return at all.
Example:
echo '{ "id": 100, "a": [{"t" : 1,"c" : 2 }] }' | jq '{t1: (.a[] | select(.t==1)).c , t2: (.a[] | select(.t==2)).c }'
Expected output:
{ "t1": 2, "t2": null }
Does anyone know how to achieve this with JQ?
EDIT:
Based on a comment made by #peak I found the solution that I was looking for.
jq -r '["id","t1","t2"],[.id, (.a[] | select(.t==1)).c//null, (.a[] | select(.t==2)).c//null ]|#csv'
The alternative operator does exactly what I was looking for.
Alternative Operator
Here's a simple solution that does not assume anything about the ordering of the items in the .a array, and easily generalizes to arbitrarily many .t values:
# Convert an array of {t, c} to a dictionary:
def tod: map({(.t|tostring): .c}) | add;
["id", "t1", "t2"], # header
(inputs
| (.a | tod) as $dict
| [.id, (range(1;3) as $i | $dict[$i|tostring]) ])
| #csv
Command-line options
Use the -n option (because inputs is being used), and the -r option (to produce CSV).
This is an absolute mess, but it works:
$ cat tmp.json
{"id": 100, "a": [{"t" : 1,"c" : 2 }, {"t": 2, "c" : 3 }] }
{"id": 200, "a": [{"t": 2, "c" : 3 }] }
{"id": 300, "a": [{"t": 1, "c" : 3 }] }
$ cat filter.jq
def t(id):
.a |
map({key: "t\(.t)", value: .c}) |
({t1:null, t2:null, id:id} | to_entries) + . | from_entries
;
inputs |
map(.id as $id | t($id)) |
(.[0] | keys) as $hdr |
([$hdr] + map(to_entries |map(.value)))[]|
#csv
$ jq -rn --slurp -f filter.jq tmp.json
"id","t1","t2"
2,3,100
,3,200
3,,300
In short, you produce a direct object containing the values from your input, then add it to a "default" object to fill in the missing keys.
Given
[{
"objects": [{
"key": "value"
},{
"key": "value"
}]
}, {
"objects": [{
"key": "value"
}, {
"key": "value"
}]
}]
How do I generate
[{
"objects": [{
"id": 0,
"key": "value"
},{
"id": 1,
"key": "value"
}]
}, {
"objects": [{
"id": 2,
"key": "value"
}, {
"id": 3,
"key": "value"
}]
}]
Using jq?
I tried to use this one, but ids are all 0:
jq '[(-1) as $i | .[] | {objects: [.objects[] | {id: ($i + 1 as $i | $i), key}]}]'
The key to a simple solution here is to break the problem down into easy pieces. This can be accomplished by defining a helper function, addId/1. Once that is done, the rest is straightforward:
# starting at start, add {id: ID} to each object in the input array
def addId(start):
reduce .[] as $o
([];
length as $l
| .[length] = ($o | (.id = start + $l)));
reduce .[] as $o
( {start: -1, answer: []};
(.start + 1) as $next
| .answer += [$o | (.objects |= addId($next))]
| .start += ($o.objects | length) )
| .answer
Inspired by #peak answer, I came up with this solution. Not much difference, just shorter way to generate IDs and opt for foreach instead of reduce since there is intermediate result involved.
def addIdsStartWith($start):
[to_entries | map((.value.id = .key + $start) | .value)];
[foreach .[] as $set (
{start: 0};
.set = $set |
.start as $start | .set.objects |= addIdsStartWith($start) |
.start += ($set.objects | length);
.set
)]