Related
TL;DR
How can I return the whole JSON after filtering inner array elements of a top-level key?
Detailed explanation
I have a JSON describing the COCO image database and it is formatted as follows (irrelevant elements truncated as ...).
{
"info": {
"description": "COCO 2017 Dataset",
...
},
"licenses": [
{
"url": "http://creativecommons.org/licenses/by-nc-sa/2.0/",
...
},
...
],
"images": [
{
"license": 4,
...
},
"annotations": [
{
"segmentation": [
[
510.66,
...
]
],
"area": 702.1057499999998,
"iscrowd": 0,
"image_id": 289343,
"bbox": [
473.07,
395.93,
38.65,
28.67
],
"category_id": 18,
"id": 1768
},
"categories": [
{
"supercategory": "person",
...
},
]
}
I need to filter annotations where category_id has one of several values, for example 1, 2.
I can successfully filter such category_ids with
jq -C ' .annotations[] | select( .category_id == 1 or .category_id == 2 ) ' instances_val2017.json | less -R
However, what is returned are only the annotations element of the total JSON as below.
{
"segmentation": [
[
162.72,
...
]
],
"area": 426.9120499999995,
"iscrowd": 0,
"image_id": 45596,
"bbox": [
161.52,
507.18,
46.45,
19.16
],
"category_id": 2,
"id": 124742
}
{
...
{
I know it's possible to return these elements as an array by wrapping the expression in [] but how can I return the entire original JSON after filtering the specified category ids?
Okay I spent 3 hours trying to solve this yesterday then this morning I posted this question and subsequently figured it out!
Here is the solution which uses the |= operator which modifies an element in place.
jq '.annotations |= map(select(.category_id | contains(1,2)))' instances_val2017.json
As per the suggestion of #peak, here is the command with == instead of contains.
jq '.annotations |= map(select(.category_id == (1,2)))' instances_val2017.json
I have an API that returns JSON - big blocks of it. Some of the key value pairs have more blocks of JSON as the value associated with a key. jq does a great job of parsing the main JSON levels. But I can't find a way to get it to 'recurse' into the values associated with the keys and pretty print them as well.
Here is the start of one of the JSON returns. Note it is only a small percent of the full return:
{
"code": 200,
"status": "OK",
"data": {
"PlayFabId": "xxxxxxx",
"InfoResultPayload": {
"AccountInfo": {
"PlayFabId": "xxxxxxxx",
"Created": "2018-03-22T19:23:29.018Z",
"TitleInfo": {
"Origination": "IOS",
"Created": "2018-03-22T19:23:29.033Z",
"LastLogin": "2018-03-22T19:23:29.033Z",
"FirstLogin": "2018-03-22T19:23:29.033Z",
"isBanned": false
},
"PrivateInfo": {},
"IosDeviceInfo": {
"IosDeviceId": "xxxxxxxxx"
}
},
"UserVirtualCurrency": {
"GT": 10,
"MB": 70
},
"UserVirtualCurrencyRechargeTimes": {},
"UserData": {},
"UserDataVersion": 15,
"UserReadOnlyData": {
"DataVersion": {
"Value": "6",
"LastUpdated": "2018-03-22T19:48:59.543Z",
"Permission": "Public"
},
"achievements": {
"Value": "[{\"id\":0,\"gamePack\":\"GAME.PACK.0.KK\",\"marblesAmount\":50,\"achievements\":[{\"id\":2,\"name\":\"Correct Round 4\",\"description\":\"Round 4 answered correctly\",\"maxValue\":10,\"increment\":1,\"currentValue\":3,\"valueUnit\":\"unit\",\"awardOnIncrement\":true,\"marbles\":10,\"image\":\"https://www.jamandcandy.com/kissinkuzzins/achievements/icons/sphinx\",\"SuccessKey\":[\"0_3_4_0\",\"0_5_4_0\",\"0_6_4_0\",\"0_7_4_0\",\"0_8_4_0\",\"0_9_4_0\",\"0_10_4_0\"],\"event\":\"Player_answered_round\",\"achieved\":false},{\"id\":0,\"name\":\"Complete
This was parsed using jq but as you can see when you get to the
"achievements": { "Vales": "[{\"id\":0,\"gamePack\":\"GAME.PACK.0.KK\",\"marblesAmount\":50,\
lq does no further parse the value at is also JSON.
Is there a filter I am missing to get it to parse the values as well as the higher level structure?
Is there a filter I am missing ...?
The filter you'll need is fromjson, but it should only be applied to the stringified JSON; consider therefore using |= as illustrated using your fragment:
echo '{"achievements": { "Vales": "[{\"id\":0,\"gamePack\":\"GAME.PACK.0.KK\",\"marblesAmount\":50}]"}}' |
jq '.achievements.Vales |= fromjson'
{
"achievements": {
"Vales": [
{
"id": 0,
"gamePack": "GAME.PACK.0.KK",
"marblesAmount": 50
}
]
}
}
recursively/1
If you want to apply fromjson recursively wherever possible, then recursively is your friend:
def recursively(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | recursively(f) )} )
elif type == "array" then map( recursively(f) )
else try (f as $f | if $f == . then . else ($f | recursively(f)) end) catch $in
end;
This would be applied as follows:
recursively(fromjson)
Example
{a: ({b: "xyzzy"}) | tojson} | tojson
| recursively(fromjson)
yields:
{
"a": {
"b": "xyzzy"
}
}
I have a json file that I need to convert to a csv file, but I am a little wary of trusting a json-to-csv converter site as the outputted data seems to be incorrect... so I was hoping to get some help here!
I have the following json file structure:
{
"GroupName": "GrpName13",
"Number": 3,
"Notes": "Test Group ",
"Units": [
{
"UnitNumber": "TestUnit13",
"DataSource": "Factory",
"ContractNumber": "TestContract13",
"CarNumber": "2",
"ControllerTypeMessageId" : 4,
"NumberOfLandings": 4,
"CreatedBy": "user1",
"CommissionModeMessageId": 2,
"Details": [
{
"DetailName": "TestFloor13",
"DetailNumber": "5"
}
],
"UnitDevices": [
{
"DeviceTypeMessageId": 1,
"CreatedBy": "user1"
}
]
}
]
}
The issue I think Im seeing is that the converters seem to not be able to comprehend the many nested data values. And the reason I think the converters are wrong is because when I try to convert back to json using them, I dont receive the same structure.
Does anyone know how to manually format this json into csv format, or know of a reliable converter than can handle nested values?
Try
www.json-buddy.com/convert-json-csv-xml.htm
if not working for you then you can try this tool
http://download.cnet.com/JSON-to-CSV/3000-2383_4-76680683.html
should be helpful!
I have tried your json on this for url:
http://www.convertcsv.com/json-to-csv.htm
As a result:
UnitNumber,DataSource,ContractNumber,CarNumber,ControllerTypeMessageId,NumberOfLandings,CreatedBy,CommissionModeMessageId,Details/0/DetailName,Details/0/DetailNumber,UnitDevices/0/DeviceTypeMessageId,UnitDevices/0/CreatedBy
TestUnit13,Factory,TestContract13,2,4,4,user1,2,TestFloor13,5,1,user1
Because it could save the path of the key,like the 'DeviceTypeMessageId' in list 'UnitDevices': it will named the columns name with 'UnitDevices/0/DeviceTypeMessageId', this could avoid the same name mistake, so you can get the columns name by its converter rules.
Hope helpful.
Here is a solution using jq
If the file filter.jq contains
def denormalize:
def headers($p):
keys_unsorted[] as $k
| if .[$k]|type == "array" then (.[$k]|first|headers("\($p)\($k)_"))
else "\($p)\($k)"
end
;
def setup:
[
keys_unsorted[] as $k
| if .[$k]|type == "array" then [ .[$k][]| setup ]
else .[$k]
end
]
;
def iter:
if length == 0 then []
elif .[0]|type != "array" then
[.[0]] + (.[1:] | iter)
else
(.[0][] | iter) as $x
| (.[1:] | iter) as $y
| [$x[]] + $y
end
;
[ headers("") ], (setup | iter)
;
denormalize | #csv
and data.json contains (note extra samples added)
{
"GroupName": "GrpName13",
"Notes": "Test Group ",
"Number": 3,
"Units": [
{
"CarNumber": "2",
"CommissionModeMessageId": 2,
"ContractNumber": "TestContract13",
"ControllerTypeMessageId": 4,
"CreatedBy": "user1",
"DataSource": "Factory",
"Details": [
{
"DetailName": "TestFloor13",
"DetailNumber": "5"
}
],
"NumberOfLandings": 4,
"UnitDevices": [
{
"CreatedBy": "user1",
"DeviceTypeMessageId": 1
},
{
"CreatedBy": "user10",
"DeviceTypeMessageId": 10
}
],
"UnitNumber": "TestUnit13"
},
{
"CarNumber": "99",
"CommissionModeMessageId": 99,
"ContractNumber": "Contract99",
"ControllerTypeMessageId": 99,
"CreatedBy": "user99",
"DataSource": "Another Factory",
"Details": [
{
"DetailName": "TestFloor99",
"DetailNumber": "99"
}
],
"NumberOfLandings": 99,
"UnitDevices": [
{
"CreatedBy": "user99",
"DeviceTypeMessageId": 99
}
],
"UnitNumber": "Unit99"
}
]
}
then the command
jq -M -r -f filter.jq data.json
will produce
"GroupName","Notes","Number","Units_CarNumber","Units_CommissionModeMessageId","Units_ContractNumber","Units_ControllerTypeMessageId","Units_CreatedBy","Units_DataSource","Units_Details_DetailName","Units_Details_DetailNumber","Units_NumberOfLandings","Units_UnitDevices_CreatedBy","Units_UnitDevices_DeviceTypeMessageId","Units_UnitNumber"
"GrpName13","Test Group ",3,"2",2,"TestContract13",4,"user1","Factory","TestFloor13","5",4,"user1",1,"TestUnit13"
"GrpName13","Test Group ",3,"2",2,"TestContract13",4,"user1","Factory","TestFloor13","5",4,"user10",10,"TestUnit13"
"GrpName13","Test Group ",3,"99",99,"Contract99",99,"user99","Another Factory","TestFloor99","99",99,"user99",99,"Unit99"
I'm trying to get the total amount for a specific value in a array.
The input would be something like
[{
"name": "mars",
"runner": [
{
"foo": null,
"idle": true
},
{
"foo": null,
"idle": true
},
{
"foo": null,
"idle": false
}
],
"name": "june",
"runner": [
{
"foo": null,
"idle": true
},
{
"foo": null,
"idle": true
},
{
"foo": null,
"idle": true
}
]
}]
The desired output
[ {"name" : "mars", "idle" : 2"},{"name" : "june", "idle" : 1"} ]
I tried using select and map, but I'm not understanding exactly how jq is working, for instance, I tried the following query
jq ' .[] | select(.runner[].idle == true) | {name: .name}'
The result was
{ "nome": "mars" } { "nome": "mars" } { "nome": "june" } {
"nome": "june" } { "nome": "june" }
(3x true in june and 2x in mars) I can keep parsing the json and get to the result I want, but it doesn't seems right.
Your input data doesn't seem to correspond exactly with your posting. I'll assume you meant:
[{"name":"mars",
"runner":[{"foo":null,"idle":true},{"foo":null,"idle":true},{"foo":null,"idle":false}]},
{"name":"june",
"runner":[{"foo":null,"idle":true},{"foo":null,"idle":true},{"foo":null,"idle":true}]}]
length
A reasonable approach would be to rely on the builtin length function:
map( { name, idle: (.runner | map(select(.idle)) | length)} )
count
A better (e.g. more efficient) solution would be to define a function that can count:
def count(s): reduce s as $i (0; .+1);
Here s is any filter that produces a stream of values. A solution to the problem at hand could then be written as follows:
map( {name, idle: count(.runner[] | select(.idle))} )
Output
The output in both cases:
[
{
"name": "mars",
"idle": 2
},
{
"name": "june",
"idle": 3
}
]
I have a fairly complex JSON data structure that I've managed to use jq to filter down to certain keys and their values. I need to combine the results though, so duplicate keys have only one array of values.
e.g.
{
"1.NBT.B": [
{
"id": 545
},
{
"id": 546
}
]
},
{
"1.NBT.B": [
{
"id": 1281
},
{
"id": 1077
}
]
}
would result in
{
"1.NBT.B": [
{
"id": 545
},
{
"id": 546
},
{
"id": 1281
},
{
"id": 1077
}
]
},
...
or even better:
[{"1.NBT.B": [545, 546, 1281, 1077]}, ...]
I need to do it without having to put in the key ("1.NBT.B") directly, since there are hundreds of these keys. I think what has me most stuck is that the objects here aren't named -- the keys are not the same between objects.
Something like this only gives me the 2nd set of ids, completing skipping the first:
reduce .[] as $item ({}; . + $item)
Part 1
The following jq function combines an array of objects in the manner envisioned by the first part of the question.
# Given an array of objects, produce a single object with an array at
# every key, the array at each key, k, being formed from all the values at k.
def merge:
reduce .[] as $o ({}; reduce ($o|keys)[] as $key (.; .[$key] += $o[$key] ));
With this definition together with the line:
merge
in a file, and with the example input modified to be a valid JSON array,
the result is:
{
"1.NBT.B": [
{
"id": 545
},
{
"id": 546
},
{
"id": 1281
},
{
"id": 1077
}
]
}
Part 2
With merge as defined above, the filter:
merge | with_entries( .value |= map(.id) )
produces:
{
"1.NBT.B": [
545,
546,
1281,
1077
]
}