I'm currently working through an issue, and can't seem to figure this one out. Here's some data so you know what I'm talking about below:
foo.json
{
"Schedule": [
{
"deviceId": 123,
"reservationId": 123456,
"username": "jdoe"
},
{
"deviceId": 456,
"reservationId": 589114,
"username": "jsmith"
}
],
"serverTime": 1522863125.019958
}
bar.json
[
{
"a": {
"b": "10.0.0.1",
"c": "hostname1"
},
"deviceId": 123
},
{
"a": {
"b": "10.0.0.2",
"c": "hostname2"
},
"deviceId": 456
}
]
foobar.json
{
"Schedule": [
{
"deviceId": 123,
"reservationId": 123456,
"username": "jdoe",
"a": {
"b": "10.0.0.1",
"c": "hostname1"
}
}
},
{
"deviceId": 456,
"reservationId": 789101,
"username": "jsmith",
"a": {
"b": "10.0.0.2",
"c": "hostname2"
}
}
],
"serverTime": 1522863125.019958
}
I'm trying to use jq to do this, and had some help from this post: https://github.com/stedolan/jq/issues/1090
The goal is to be able to combine JSON, using some key as a common point between the documents. The data may be nested any amount of levels.. In this case foo.json has nested data only two levels deep, but needs to be combined with data nested 1 level deep.
Any and all suggestions would be super helpful. I'm also happy to clarify and answer questions if needed. Thank you!
With foobar.jq as follows:
def dict(f):
reduce .[] as $o ({}; .[$o | f | tostring] = $o ) ;
($bar | dict(.deviceId)) as $dict
| .Schedule |= map(. + ($dict[.deviceId|tostring] ))
the invocation:
jq -f foobar.jq --argfile bar bar.json foo.json
yields the output shown below.
Notice that the referents in the dictionary contain the full object (including the key/value pair for "deviceId"), but it's not necessary to del(.deviceId) because of the way + is defined in jq.
Output
{
"Schedule": [
{
"deviceId": 123,
"reservationId": 123456,
"username": "jdoe",
"a": {
"b": "10.0.0.1",
"c": "hostname1"
}
},
{
"deviceId": 456,
"reservationId": 589114,
"username": "jsmith",
"a": {
"b": "10.0.0.2",
"c": "hostname2"
}
}
],
"serverTime": 1522863125.019958
}
Related
I want create a more simple json with the same original structure but with one a small sample.
As example, If I have this json:
{
"field1": [
{
"a": "F1A1",
"b": "F1B1"
},
{
"a": "F1A2",
"b": "F1B2"
},
{
"a": "F1A3",
"b": "F1B3"
},
{
"a": "F1A4",
"b": "F1B4"
}
],
"field2": [
{
"a": "F2A1",
"b": "F2B1"
},
{
"a": "F2A2",
"b": "F2B2"
}
],
"field3": [
{
"a": "F3A1",
"b": "F3B1"
},
{
"a": "F3A2",
"b": "F3B2"
}
]
}
I want to get the first array element from the first field. So I was expecting this:
{
"field1": [
{
"a": "F1A1",
"b": "F1B1"
}
],
}
I executed jq "select(.field1[0])" tmp.json but it returns the original json.
Bonus:
As bonus, how to do the same but extracting let's say field1 and elements in the array with a=="F1A1" and a=="F1A4", so will expect?:
{
"field1": [
{
"a": "F1A1",
"b": "F1B1"
},
{
"a": "F1A4",
"b": "F1B4"
}
]
}
reduce the oouter object to your field using {field1}, then map this field to an array containing only the first item:
jq '{field1} | map_values([first])'
{
"field1": [
{
"a": "F1A1",
"b": "F1B1"
}
]
}
To filter for certain items use select:
jq '{field1} | map_values(map(select(.a == "F1A1" or .a == "F1A4")))'
{
"field1": [
{
"a": "F1A1",
"b": "F1B1"
},
{
"a": "F1A4",
"b": "F1B4"
}
]
}
As you can see, select does something different. It passes on its input if the argument evaluates to true. Therefore its output is either all or nothing, never just a filtered part. (Of course, you can use select to achieve specific filtering, as shown above.)
corresponding to jq ~ is there a better way to collapse single object arrays? and R: Nested data.table to JSON
how do I collapse only specific elements?
I want to get rid of the "group" arrays in
[
{
"id2": "A",
"group": [
{
"data": [
{
"id1": 1,
"group": [
{
"data": [
{
"a": 1,
"b": 1
},
{
"a": 2,
"b": 2
}
],
"type": "test"
}
],
"type": "B"
}
],
"type": "C"
}
]
},
{
"id2": "C",
"group": [
{
"data": [
{
"id1": 3,
"group": [
{
"data": [
{
"a": 1,
"b": 1
}
],
"type": "test"
}
],
"type": "B"
}
],
"type": "C"
}
]
}
]
desired output
[{
"id2": "A",
"group": {
"data": [{
"id1": 1,
"group": {
"data": [{
"a": 1,
"b": 1
},
{
"a": 2,
"b": 2
}
],
"type": "test"
},
"type": "B"
}],
"type": "C"
}
},
{
"id2": "C",
"group": {
"data": [{
"id1": 3,
"group": {
"data": [{
"a": 1,
"b": 1
}],
"type": "test"
},
"type": "B"
}],
"type": "C"
}
}
]
The line 'walk(if type=="array" and length==1 then .[0] else . end)' additionally removes the array from the single "data" object.
Unfortunately, we are not able to install the jq 1.6 version on our RStudio Server und thereby I'm not able to use the walk function. (Although is working perfectly fine on my local system)
Can anybody help me out with an alternative solution without walk? Would be highly appreciated.
edit
Ok I got it. I can manually add the walk function such as:
'def walk(f):
. as $in
| if type == "object" then
reduce keys_unsorted[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end; walk(if type=="object"
and has("group")
and (.group | type)=="array"
and (.group | length)==1
then .group = .group[0]
else . end)'
We could operate one level higher in the nesting hierarchy, and test for "group" being a key, then update accordingly .group = .group[0] instead of . = .[0]
jq 'walk(if type=="object"
and has("group")
and (.group | type)=="array"
and (.group | length)==1
then .group = .group[0]
else . end)'
I have been playing around with jq to format a json file but I am having some issues trying to solve a particular transformation. Given a test.json file in this format:
[
{
"name": "A", // This would be the first key
"number": 1,
"type": "apple",
"city": "NYC" // This would be the second key
},
{
"name": "A",
"number": "5",
"type": "apple",
"city": "LA"
},
{
"name": "A",
"number": 2,
"type": "apple",
"city": "NYC"
},
{
"name": "B",
"number": 3,
"type": "apple",
"city": "NYC"
}
]
I was wondering, how can I format it this way using jq?
[
{
"key": "A",
"values": [
{
"key": "NYC",
"values": [
{
"number": 1,
"type": "a"
},
{
"number": 2,
"type": "b"
}
]
},
{
"key": "LA",
"values": [
{
"number": 5,
"type": "b"
}
]
}
]
},
{
"key": "B",
"values": [
{
"key": "NYC",
"values": [
{
"number": 3,
"type": "apple"
}
]
}
]
}
]
I have followed this thread Using jq, convert array of name/value pairs to object with named keys and tried to group the json using this expression
jq '. | group_by(.name) | group_by(.city) ' ./test.json
but I have not been able to add the keys in the output.
You'll want to group the items at the different levels and building out your result objects as you want.
group_by(.name) | map({
key: .[0].name,
values: (group_by(.city) | map({
key: .[0].city,
values: map({number,type})
}))
})
Just keep in mind that group_by/1 yields groups in a sorted order. You'll probably want an implementation that preserves that order.
def group_by_unsorted(key_selector):
reduce .[] as $i ({};
.["\($i|key_selector)"] += [$i]
)|[.[]];
I need to filter a JSON with a nested strucutre like below.
All objects in array b where attribute x contains a "z" in the value of x should be filtered out. The rest should stay in the file.
{
"a": {
"b": [
{
"c": "1",
"x": "aaa",
},
{
"c": "2",
"x": "aza",
},
{
"c": "7",
"x": "azb",
}
]
},
"d": {
"e": [
"1"
],
"f": [
"2"
]
}
}
Expected output:
{
"a": {
"b": [
{
"c": "1",
"x": "aaa"
}
]
},
"d": {
"e": [
"1"
],
"f": [
"2"
]
}
}
use select with contains:
jq '.a.b|=[.[]|select(.x|contains("z")|not)]' file
I want to convert a complex JSON file into a simple JSON file using JQ. However, the query I'm using generates an incorrect output.
My (cut down) JSON file:
[
{
"id": 100,
"foo": [
{
"bar": [
{"type": "read"},
{"type": "write"}
],
"users": ["admin_1"],
"groups": []
},
{
"bar": [
{"type": "execute"},
{ "type": "read"}
],
"users": [],
"groups": ["admin_2"]
}
]
},
{
"id": 101,
"foo": [
{
"bar": [
{"type": "read"}
],
"users": [
"admin_3"
],
"groups": []
}
]
}
]
I need to generate a flatter JSON file and combine the users and groups into one field, similar to this:
[
{
"id": 100,
"users_groups": [
"admin_1",
"admin_2"
],
"bar": ["read"]
},
{
"id": 100,
"users_groups": ["admin_1"],
"bar": ["write"]
},
{
"id": 100,
"users_groups": ["admin_2"],
"bar": ["execute"]
},
{
"id": 101,
"users_groups": ["admin_3"],
"bar": ["read"]
}
]
Everything I try in JQ results in me getting an incorrect output (where admin_1 incorrectly has bar=execute and admin_2 incorrectly has bar=write), similar to the following:
[
{
"id": 100,
"users_groups": [
"admin_1",
"admin_2"
],
"bar": ["read", "write", "execute"]
},
{
"id": 101,
"users_groups": ["admin_3"],
"bar": ["read"]
}
]
I have tried many vairiats of this query - any idea what I should be doing instead?
cat file.json | jq -r '[.[] | select(has("foo")) |{"id", "users":(.foo[] | .users), "groups":(.foo[] | .groups), "bar":([.foo[].bar[] | .type])} ] '
The following filter groups by "type" as the question seems to require:
map(.id as $id
| [.foo[]
| {id: $id, bar: .bar[].type} +
{"users_groups": (.users + .groups)[]} ]
| group_by(.bar)
| map(.[0] + {"users_groups": [.[].users_groups]}) )
Output
[
[
{
"id": 100,
"bar": "execute",
"users_groups": [
"admin_2"
]
},
{
"id": 100,
"bar": "read",
"users_groups": [
"admin_1",
"admin_2"
]
},
{
"id": 100,
"bar": "write",
"users_groups": [
"admin_1"
]
}
],
[
{
"id": 101,
"bar": "read",
"users_groups": [
"admin_3"
]
}
]
]
Variations
To achieve the array-of-objects output format, simply tack on | [.[][]];
it would similarly be trivially easy to ensure that .bar is array-valued, though that might be pointless given that the grouping is by .type.