Processing JSON with jq - handling array index/name into output - json

I'm trying to use jq to parse a JSON file for me. I want to get a value from a definition header into the output data in place of an index. A simplified example:
{
"header": {
"type": {
"0": {
"name": "Cats"
},
"3": {
"name": "Dogs"
}
}
},
"data": [
{
"time": "2019-01-01T02:00:00Z",
"reading": {
"0": {"value": 90, "note": "start" },
"3": {"value": 100 }
}
}
]
}
Using a jq command like jq '.data[] | {time: .time, data: .reading[]}' gives me:
"time": "2019-01-01T02:00:00Z",
"data": {
"value": 90,
"note": "start"
}
}
{
"time": "2019-01-01T02:00:00Z",
"data": {
"value": 100
}
}
I need to get "Cats" or "Dogs" into the result, heading towards an SQL insert.
Something like:
{
"time": "2019-01-01T02:00:00Z",
"data": {
"type: "Cats", <- line added
"value": 90,
"note": "start"
}
}
...
Or better yet:
{
"time": "2019-01-01T02:00:00Z",
"Cats": { <- label set to "Cats" instead of "data"
"value": 90,
"note": "start"
}
}
...
Is there a way I can get - what I see as the array index "0" or "3" - to be added as "Cats" or "Dogs"?

Using the built-in function, INDEX, for creating a dictionary allows a straightforward solution as follows:
(.header.type
| INDEX(to_entries[]; .key)
| map_values(.value.name)) as $dict
| .data[]
| (.reading | keys_unsorted[]) as $k
| {time} + { ($dict[$k]) : .reading[$k] }
Output
{
"time": "2019-01-01T02:00:00Z",
"Cats": {
"value": 90,
"note": "start"
}
}
{
"time": "2019-01-01T02:00:00Z",
"Dogs": {
"value": 100
}
}

Related

Using jq to search for a value based on a key located deep in json file

I am new to jq and I'm trying to use it to search for a value in a json file based on a key that is located deep in the json structure. Here is a sample of my json file:
{
"data": {
"inventory": {
"location": "remote",
"list": {
"content": [
{
"item": {
"name": "minivan"
},
"owner": {
"id": "12345",
"state": "CA"
}
},
{
"item": {
"name": "sedan"
},
"owner": {
"id": "67890",
"state": "AZ"
}
}
]
}
}
}
}
An example of search that I'm trying to do is:
select item.name where owner.id = "67890"
and the expected output would be:
item.name = "sedan"
I'm trying to run the following:
jq '.[] | select .owner.id = "67890" | .item.name' json
and it generates an error:
jq: error: select/0 is not defined at <top-level>, line 1:
.[] | select .owner.id = "67890" | .item.name
jq: 1 compile error
Any pointers on how to do this in jq would be much appreciated!
Thanks!
First, you have to "navigate" to where you want to make the query. This seems to be an array.
.data.inventory.list.content
[
{
"item": {
"name": "minivan"
},
"owner": {
"id": "12345",
"state": "CA"
}
},
{
"item": {
"name": "sedan"
},
"owner": {
"id": "67890",
"state": "AZ"
}
}
]
Demo
Next, let's iterate over that array's items, which gives us a stream of objects.
.[]
{
"item": {
"name": "minivan"
},
"owner": {
"id": "12345",
"state": "CA"
}
}
{
"item": {
"name": "sedan"
},
"owner": {
"id": "67890",
"state": "AZ"
}
}
Demo
From these objects we select those that match your criteria.
select(.owner.id == "67890")
{
"item": {
"name": "sedan"
},
"owner": {
"id": "67890",
"state": "AZ"
}
}
Demo
Finally, we extract the value you're interested in.
.item.name
"sedan"
Demo
Everything combined in a jq call would be:
jq '.data.inventory.list.content[] | select(.owner.id == "67890").item.name'
"sedan"
Demo
This output is still valid JSON document (containing nothing but a JSON string). If you want to process the output as raw text, use the --raw-output (or -r) option:
jq -r '.data.inventory.list.content[] | select(.owner.id == "67890").item.name'
sedan
Demo
Here's a solution that avoids having to "navigate" to the right place, and which is also quite close to your SQL-like query:
..
| objects
| select(.owner and
(.owner|type=="object" and .id == "67890"))
.item.name
or more succinctly:
..|objects|select(.owner.id? == "67890").item.name

Storing top level JSON key for use after reshaping JSON with jq

I'm playing around with some MTGJSON data and I'm trying to convert data from a file called AllPrintings.json that looks like:
{
"10E": {
"cards": [
{
"name": "Abundance",
"prices": {
"paper": {
"2020-06-11": 1.4
},
"paperFoil": {
"2020-06-11": 31.12
}
},
"uuid": "1669af17-d287-5094-b005-4b143441442f"
},
{
"name": "Academy Researchers",
"prices": {
"paper": {
"2020-06-11": 0.36
},
"paperFoil": {
"2020-06-11": 1.22
}
},
"uuid": "047d5499-a21c-5f5c-9679-1599fcaf9815"
}
]
},
"BFZ": {
"cards": [
{
"name": "Adverse Conditions",
"prices": {
"paper": {
"2020-06-11": 0.23
},
"paperFoil": {
"2020-06-11": 1.86
}
},
"uuid": "1669af17-d287-5094-b005-4b143441123"
},
{
"name": "Akoum Firebird",
"prices": {
"paper": {
"2020-06-11": 0.51
},
"paperFoil": {
"2020-06-11": 3.85
}
},
"uuid": "047d5499-a21c-5f5c-9679-1599fcafad567"
}
]
}
}
Into:
{
{
"name": "Abundance",
"price": 1.4,
"uuid": "1669af17-d287-5094-b005-4b143441442f",
"set": "10E"
},
{
"name": "Academy Researchers",
"price": 0.36,
"uuid": "047d5499-a21c-5f5c-9679-1599fcaf9815",
"set": "10E"
},
{
"name": "Adverse Conditions",
"price": 0.23,
"uuid": "1669af17-d287-5094-b005-4b143441123",
"set": "BFZ"
},
{
"name": "Akoum Firebird",
"price": 0.51,
"uuid": "047d5499-a21c-5f5c-9679-1599fcafad567",
"set": "BFZ"
},
}
I'm able to get everything except the set by running
cat AllPrintings.json | jq '[.[] | .cards | .[] | {uuid: .uuid, name: .name, price: .prices.paper | .[]? }]'
which returns
{
{
"name": "Abundance",
"price": 1.4,
"uuid": "1669af17-d287-5094-b005-4b143441442f",
},
{
"name": "Academy Researchers",
"price": 0.36,
"uuid": "047d5499-a21c-5f5c-9679-1599fcaf9815",
},
{
"name": "Adverse Conditions",
"price": 0.23,
"uuid": "1669af17-d287-5094-b005-4b143441123",
},
{
"name": "Akoum Firebird",
"price": 0.51,
"uuid": "047d5499-a21c-5f5c-9679-1599fcafad567",
},
}
I've tried storing the top level keys as $k and can get an array of the keys in a separate command but I'm unable to keep iterating over the original data afterwards. I've tried the comma separator but I get errors or the query hangs. For instance
cat AllPrintings.json | jq ‘. | keys as $k, [.[] | .cards | .[] | {uuid: .uuid, name: .name, price: .prices.paper, key: $k | .[]? }]'
I've search here and have read through the jq documentation but I'm not sure exactly what I'm looking for. I'm also likely overthinking this or missing an obvious solution. Any help would be appreciated. If this is a duplicate question please link me to the original and I'll delete my post.
Thanks.
You were close. The missing part is to_entries, which converts an object into {key, value} pairs:
to_entries | map(.key as $set | .value.cards[] | {uuid, name, price: .prices.paper | .[]?, set: $set })

jq summarize similar elements

I want to use jq to summarize elements with the same description field value.
So that each unique field value of description gets it's own element with the amount field summarized.
I'm using jq 1.5
Before:
{ "frames":
[
{ "description": "Stuff", "amount": 8 },
{ "description": "Stuff", "amount": 4 },
{ "description": "other_stuff", "amount": 2 },
{ "description": "more_stuff", "amount": 20 }
]
}
After:
{ "frames":
[
{ "description": "Stuff": 12 },
{ "description": "other_stuff", "amount": 2 },
{ "description": "more_stuff", "amount": 20 }
]
}
With your input, the following program produces valid JSON in the spirit of the question:
def sum_by(f;g): reduce .[] as $x ({}; .[$x|f] += ($x|g));
.frames |= sum_by(.description; .amount)
Using sum_by is more efficient than using group_by for various reasons.
Output
{
"frames": {
"Stuff": 12,
"other_stuff": 2,
"more_stuff": 20
}
}
Variant
If you want description and amount in the output, you could tweak the above as follows:
.frames |=
(sum_by(.description; .amount)
| to_entries
| map( {description: .key, amount: .value} ))
The output would then be:
{
"frames": [
{
"description": "Stuff",
"amount": 12
},
{
"description": "other_stuff",
"amount": 2
},
{
"description": "more_stuff",
"amount": 20
}
]
}

Getting unique values from nested Array using jq

Trying to get unique values stored in items array for each group. somehow it's always mixed...
My JSON looks like this:
{
"start": 1534425916,
"stop": 1535030716,
"groups": [
{
"group": "transmission",
"data": {
"events": 665762,
},
"items": [
{
"item": "manualni",
"data": {
"events": 389158,
}
},
{
"item": "automaticka",
"data": {
"events": 276604,
}
}
]
},
{
"group": "vat",
"data": {
"events": 671924,
},
"items": [
{
"item": "ne",
"data": {
"events": 346221,
}
},
{
"item": "ano",
"data": {
"events": 325703,
}
}
]
}
]
}
Desired result is the following:
{
"id": "transmission",
"value": [
"manualni",
"automaticka",
]
}
{
"id": "vat",
"value": [
"ne",
"ano"
]
}
Tried with this filter on command line:
| jq '{id: .groups[].group, value: [.groups[].items[].item]}'
Which results in the above mentioned mixed up result:
{
"id": "transmission",
"value": [
"manualni",
"automaticka",
"ne",
"ano"
]
}
{
"id": "vat",
"value": [
"manualni",
"automaticka",
"ne",
"ano"
]
}
Any idea how to receive the uniquified values here? Thanks in advance!
This gets the desired result. I think the manual entry under .[] explains why it works.
jq '.groups[] | {"id": .group, "value": [.items[].item]}'

jq select dynamic items from json

I have JSON like this:
{
"photo_807": "Ih2RnaBTg2o.jpg",
"photo_604": "zodCm9fQgX8.jpg",
"photo_130": "4Dx-SUNKBw4.jpg",
"photo_75": "7COWb8ou1qA.jpg",
"user_id": 100,
"owner_id": -2435432542783750,
"access_key": "fc5275423676514042234324265cc3df7607c",
"post_id": 380435645368865101,
"date": 14858616848616779856424245814,
"text": "",
"height": 417,
"width": 740,
"id": 45624575446886886564368555,
"album_id": -36
}
I want to get only Photo values, from output i want get this:
"photo_807": "Ih2RnaBTg2o.jpg"
"photo_604": "zodCm9fQgX8.jpg"
"photo_130": "4Dx-SUNKBw4.jpg"
Now about my problem, from next JSON file i will get photo items with new names like this:
"photo_181": "Ih2RnaBTg2o.jpg",
"photo_583": "zodCm9fQgX8.jpg",
"photo_975": "4Dx-SUNKBw4.jpg",
"photo_32": "7COWb8ou1qA.jpg",
How I can get this values from dynamic items photo_* ?
I try something like this:
cat ./json3.txt | jq '.response.items[].attachments[].photo | select(.photo | startswith("photo"))'
But it doesn't work.
When I run :
cat ./json3.txt | jq '.response.items[].attachments[].photo'
I got this output with all items:
{
"photo_807": "Ih2RnaBTg2o.jpg",
"photo_604": "zodCm9fQgX8.jpg",
"photo_130": "4Dx-SUNKBw4.jpg",
"photo_75": "7COWb8ou1qA.jpg",
"user_id": 100,
"owner_id": -2435432542783750,
"access_key": "fc5275423676514042234324265cc3df7607c",
"post_id": 380435645368865101,
"date": 14858616848616779856424245814,
"text": "",
"height": 417,
"width": 740,
"id": 45624575446886886564368555,
"album_id": -36
}
Can someone help me?
Thanks in advance!
You can use a jq filter as below as tested on jq-play!
jq '. | to_entries[] | select(.key | startswith("photo")) | "\(.key) :\(.value)"' json
"photo_807 :Ih2RnaBTg2o.jpg"
"photo_604 :zodCm9fQgX8.jpg"
"photo_130 :4Dx-SUNKBw4.jpg"
"photo_75 :7COWb8ou1qA.jpg"
The idea is to use the to_entries[] built-in, which converts your input into a key-value pair as below. See below the output of just using to_entries[]
jq '. | to_entries[]' json
{
"key": "photo_807",
"value": "Ih2RnaBTg2o.jpg"
}
{
"key": "photo_604",
"value": "zodCm9fQgX8.jpg"
}
{
"key": "photo_130",
"value": "4Dx-SUNKBw4.jpg"
}
{
"key": "photo_75",
"value": "7COWb8ou1qA.jpg"
}
{
"key": "user_id",
"value": 100
}
{
"key": "owner_id",
"value": -2435432542783750
}
{
"key": "access_key",
"value": "fc5275423676514042234324265cc3df7607c"
}
{
"key": "post_id",
"value": 380435645368865100
}
{
"key": "date",
"value": 14858616848616779000000000000
}
{
"key": "text",
"value": ""
}
{
"key": "height",
"value": 417
}
{
"key": "width",
"value": 740
}
{
"key": "id",
"value": 45624575446886885000000000
}
{
"key": "album_id",
"value": -36
}
On this output, we are filtering on the .key value which starts with your string, photo in your case, using the built-in startswith(), and printing the both the .key and .value pair for the matching objects.