I use jq 1.6 in a Windows 10 PowerShell enviroment and trying to select keys from coincidentally numeric json objects.
Json exampel:
{
"alliances_info":{
"744085325458334213":{
"emblem":3,
"name":"wellwell",
"member_count":1,
"level":1,
"military_might":1035,
"public":false,
"tag":"MELL",
"slogan":"",
"id":744085325458334213
},
"744128593839677958":{
"emblem":0,
"name":"Brave",
"member_count":1,
"level":1,
"military_might":1035,
"public":false,
"tag":"GABA",
"slogan":"",
"id":744128593839677958
},
"746034084459209223":{
"emblem":0,
"name":"Queen",
"member_count":1,
"level":1,
"military_might":1035,
"public":false,
"tag":"QUE",
"slogan":"",
"id":746034084459209223
},
"750446471312466445":{
"emblem":0,
"name":"Phoenix Inc",
"member_count":35,
"level":6,
"military_might":453369,
"public":true,
"tag":"PHOI",
"slogan":"",
"id":750446471312466445
},
"750446518934594062":{
"emblem":11,
"name":"Australia",
"member_count":44,
"level":8,
"military_might":957211,
"public":true,
"tag":"AUST",
"slogan":"Go Australia",
"id":750446518934594062
}
},
"server_version":"v7.190.4-master.000000006"
}
I tried several jq commands:
.alliances_info | .[] | [{alliance_name: .name, alliance_count: .member_count, alliance_level: .level, alliance_power: .military_might, alliance_tag: .tag, alliance_slogan: .slogan, alliance_id: .id}]
or
.alliances_info | .. | objects | [{alliance_name: .name, alliance_c
ount: .member_count, alliance_level: .level, alliance_power: .military_might, alliance_tag: .tag, alliance_slogan: .slog
an, alliance_id: .id}]
But Always get a jq error: parse error: Invalid numeric literal at line 1, column 3
I renounce on the object Building in the first command (and built only a Array) it works. But i need that objects. Any tips?
BR
Timo
Your first query works perfectly well with the given JSON sample. Perhaps you're invoking jq incorrectly. If you have the jq program in a file, say select.jq, you'd invoke jq like so:
jq -f select.jq sample.json
If that doesn't help, then try:
jq empty sample.json
If that fails, there might be something wrong with the encoding of the JSON.
I'm not sure I understand what you want.
Your first attempt works for me, but generates one output for JSON value in the input. That is, I created a file named so.json and put in it your JSON from above:
{
"alliances_info": {
"744085325458334213": {
"emblem": 3,
⋮
}
When I run your program , I get:
$ jq '.alliances_info | .[] | [{alliance_name: .name, alliance_count: .member_count, alliance_level: .level, alliance_power: .military_might, alliance_tag: .tag, alliance_slogan: .slogan, alliance_id: .id}]' so.json
[
{
"alliance_name": "wellwell",
"alliance_count": 1,
"alliance_level": 1,
"alliance_power": 1035,
"alliance_tag": "MELL",
"alliance_slogan": "",
"alliance_id": 744085325458334200
}
]
[
{
"alliance_name": "Brave",
⋮
]
If you want an array at all, you probably want one array containing all the alliances like this:
$ jq '.alliances_info | [ .[] | { alliance_name: .name, alliance_id: .id } ]' so.json
[
{
"alliance_name": "wellwell",
"alliance_id": 744085325458334200
},
{
"alliance_name": "Brave",
"alliance_id": 744128593839678000
},
{
"alliance_name": "Queen",
"alliance_id": 746034084459209200
},
{
"alliance_name": "Phoenix Inc",
"alliance_id": 750446471312466400
},
{
"alliance_name": "Australia",
"alliance_id": 750446518934594000
}
]
Starting from the left,
- .alliances_info looks in its input object for the field named "alliances_info" and outputs its value
- the | next says take the output from the left-hand side and pass those as inputs to the right-hand side.
- right after that first |, I have a [ «jq expressions» ] which tells jq to create one JSON array output for each input; the elements of that array are the outputs of that inner «jq expressions»
- that inner expression starts with .[] which means to produce one output for each JSON value (ignoring the keys) in the input object. For us, that will be the objects named "744085325458334213", "744128593839677958", …
- The next | uses those objects as input and for each, generates a JSON object { alliance_name: .name, alliance_id: .id }
That's why I end up with one JSON array containing 5 JSON objects.
As far as I can tell, you are mostly just renaming a bunch of the fields. For that, you could just do something like this:
$ jq --argjson renameMap '{ "name": "alliance_name", "member_count": "alliance_count", "level": "alliance_level", "military_might": "alliance_power", "tag": "alliance_tag", "slog": "alliance_slogan"}' '.alliances_info |= ( . | [ to_entries[] | ( .value |= ( . | [ to_entries[] | ( .key |= ( if $renameMap[.] then $renameMap[.] else . end ) ) ] | from_entries ) ) ] | from_entries )' so.json
{
"alliances_info": {
"744085325458334213": {
"emblem": 3,
"alliance_name": "wellwell",
"alliance_count": 1,
"alliance_level": 1,
"alliance_power": 1035,
"public": false,
"alliance_tag": "MELL",
"slogan": "",
"id": 744085325458334200
},
"744128593839677958": {
"emblem": 0,
"alliance_name": "Brave",
"alliance_count": 1,
"alliance_level": 1,
"alliance_power": 1035,
"public": false,
"alliance_tag": "GABA",
"slogan": "",
"id": 744128593839678000
},
⋮
},
"server_version": "v7.190.4-master.000000006"
}
well i am a idiot (to be here totally clear). I found the reason (and this is normally a nobrainer...). I read the input from a file and the funny thing is that the file is Unicode but no UTF8. after recoding the command is working fine. Thanks for the help.
BR
Timo
Related
I've got some JSON like the following (I've filtered the output here):
[
{
"Tags": [
{
"Key": "Name",
"Value": "example1"
},
{
"Key": "Irrelevant",
"Value": "irrelevant"
}
],
"c7n:MatchedFilters": [
"tag: example_tag_rule"
],
"another_key": "another_value_I_dont_want"
},
{
"Tags": [
{
"Key": "Name",
"Value": "example2"
}
],
"c7n:MatchedFilters": [
"tag:example_tag_rule",
"tag: example_tag_rule2"
]
}
]
I'd like to create a csv file with the value within the Name key and all of the "c7n:MatchedFilters" in the array. I've made a few attempts but still can't get quite the output I expect. There's some example code and the output below:
#Prints the key that I'm after.
cat new.jq | jq '.[] | [.Tags[], {"c7n:MatchedFilters"}] | .[] | select(.Key=="Name")|.Value'
"example1"
"example2"
#Prints all the filters in an array I'm after.
cat new.jq | jq -r '.[] | [.Tags[], {"c7n:MatchedFilters"}] | .[] | select(."c7n:MatchedFilters") | .[]'
[
"tag: example_tag_rule"
]
[
"tag:example_tag_rule",
"tag: example_tag_rule2"
]
#Prints *all* the tags (including ones I don't want) and all the filters in the array I'm after.
cat new.jq | jq '.[] | [.Tags[], {"c7n:MatchedFilters"}] | select((.[].Key=="Name") and (.[]."c7n:MatchedFilters"))'
[
{
"Key": "Name",
"Value": "example1"
},
{
"Key": "Irrelevant",
"Value": "irrelevant"
},
{
"c7n:MatchedFilters": [
"tag: example_tag_rule"
]
}
]
[
{
"Key": "Name",
"Value": "example2"
},
{
"c7n:MatchedFilters": [
"tag:example_tag_rule",
"tag: example_tag_rule2"
]
}
]
I hope this makes sense, let me know if I've missed anything.
Your attempts are not working because you start out with [.Tags[], {"c7n:MatchedFilters"}] to construct one array containing all the tags and an object containing the filters. You are then struggling to find a way to process this entire array at once because it jumbles together these unrelated things without any distinction. You will find it much easier if you don't combine them in the first place!
You want to find the single tag with a Key of "Name". Here's one way to find that:
first(
.Tags[]|
select(.Key=="Name")
).Value as $name
By using a variable binding we can save it for later and worry about constructing the array separately.
You say (in the comments) that you just want to concatenate the filters with spaces. You can do that easily enough:
(
."c7n:MatchedFilters"|
join(" ")
) as $filters
You can combine all this together like follows. Note that each variable binding leaves the input stream unchanged, so it's easy to compose everything.
jq --raw-output '
.[]|
first(
.Tags[]|
select(.Key=="Name")
).Value as $name|
(
."c7n:MatchedFilters"|
join(" ")
) as $filters|
[$name, $filters]|
#csv
Hopefully that's easy enough to read and separates out each concept. We break up the array into a stream of objects. For each object, we find the name and bind it to $name, we concatenate the filters and bind them to $filters, then we construct an array containing both, then we convert the array to a CSV string.
We don't need to use variables. We could just have a big array constructor wrapped around the expression to find the name and the expression to find the filters. But I hope you can see the variables make things a bit flatter and easier to understand.
I have a response from curl in a format like this:
[
{
"list": [
{
"value": 1,
"id": 12
},
{
"value": 15,
"id": 13
},
{
"value": -4,
"id": 14
}
]
},
...
]
Given a mapping between ids like this:
{
"12": "newId1",
"13": "newId2",
"14": "newId3"
}
I want to make this:
[
{
"list": {
"newId1": 1,
"newId2": 15,
"newId3": -4,
}
},
...
]
Such that I get a mapping from ids to values (and along the way I'd like to remap the ids).
I've been working at this for a while and every time I get a deadend.
Note: I can use Shell or the like to preform loops if necessary.
edit: Here's one version what I've developed so far:
jq '[].list.id = ($mapping.[] | select(.id == key)) | del(.id)' -M --argjson "mapping" "$mapping"
I don't think it's the best one, but I'm looking to see if I can find an old version that was closer to what I need.
[EDIT: The following response was in answer to the question when it described (a) the mapping as shown below, and (b) the input data as having the form:
[
{
"list": [
{
"value": 1,
"id1": 12
},
{
"value": 15,
"id2": 13
},
{
"value": -4,
"id3": 14
}
]
}
]
END OF EDIT]
In the following I'll assume that the mapping is available via the following function, but that is an inessential assumption:
def mapping: {
"id1": "newId1",
"id2": "newId2",
"id3": "newId3"
} ;
The following jq filter will then produce the desired output:
map( .list
|= (map( to_entries[]
| (mapping[.key]) as $mapped
| select($mapped)
| {($mapped|tostring): .value} )
| add) )
There's plenty of ways to skin a cat. I'd do it like this:
.[].list |= reduce .[] as $i ({};
($i.id|tostring) as $k
| (select($mapping | has($k))[$mapping[$k]] = $i.value) // .
)
You would just provide the mapping through a separate file or argument.
$ cat program.jq
.[].list |= reduce .[] as $i ({};
($i.id|tostring) as $k
| (select($mapping | has($k))[$mapping[$k]] = $i.value) // .
)
$ cat mapping.json
{
"12": "newId1",
"13": "newId2",
"14": "newId3"
}
$ jq --argfile mapping mapping.json -f program.jq input.json
[
{
"list": {
"newId1": 1,
"newId2": 15,
"newId3": -4
}
}
]
Here is a reduce-free solution to the revised problem.
In the following I'll assume that the mapping is available via the following function, but that is an inessential assumption:
def mapping:
{
"12": "newId1",
"13": "newId2",
"14": "newId3"
} ;
map( .list
|= (map( mapping[.id|tostring] as $mapped
| select($mapped)
| {($mapped): .value} )
| add) )
The "select" is for safety (i.e., it checks that the .id under consideration is indeed mapped). It might also be appropriate to ensure that $mapped is a string by writing {($mapped|tostring): .value}.
I have some JSON like this:
{
"x": [
{
"name": "Hello",
"id": "211"
},
{
"name": "Goodbye",
"id": "221"
},
{
"name": "Christmas",
"id": "171"
}
],
"y": "value"
}
Using jq, given a name value (e.g. Christmas) how can I get it's associated id (i.e. 171).
I've got as far as being able to check for presence of the name in one of the array's objects, but I can't work out how to filter it down
jq -r 'select(.x[].name == "Christmas")'
jq approach:
jq -r '.x[] | select(.name == "Christmas").id' file
171
The function select(boolean_expression) produces its input unchanged if boolean_expression returns true for that input, and produces no output otherwise.
It can also been done like:
jq '.x[] | select(.name == "Christmas").id'
Also you can try this at link online jq play
I have the following JSON data:
{
"status": "ok",
"ok": true,
"data": "MFR-L",
"stores": [{
"name": "KOLL",
"lat": 52.93128,
"lng": 6.962956,
"dist": 1,
"x10": 1.129,
"isOpen": true
},
{
"name": "Takst",
"lat": 52.9523773,
"lng": 6.981644,
"dist": 1.3,
"x10": 1.809,
"isOpen": false
}]
}
I'm trying to convert it to a flat file using JQ, but I keep running into all sorts of problems, especially because of the file types ("cannot index boolean with string", etc).
This post has helped me flatten the contents of the array so far, like this:
jq -r -s 'map(.stores | map({nm: .name, lt: .lat} | [.nm, .lt])) | add [] | #csv
How can I get the contents higher up in the hierarchy to map to the array contents?
You could always collect the values you want from the parent objects separately from the child objects and combine them later.
e.g.,
$ jq -r '[.data] + (.stores[] | [.name, .lat, .lng, .dist]) | #csv' input.json
yields
"MFR-L","KOLL",52.93128,6.962956,1
"MFR-L","Takst",52.9523773,6.981644,1.3
There are several ways in which the illustrative JSON might be "flattened" (e.g. to CSV), but the following two approaches may be of interest. (I've omitted the invocation of #csv for ease-of-reading.)
$ jq '[.data, .stores[][]]' in.json
[
"MFR-L",
"KOLL",
52.93128,
6.962956,
1,
1.129,
true,
"Takst",
52.9523773,
6.981644,
1.3,
1.809,
false
]
$ jq '.data as $data | .stores[] | [$data, .[]]' in.json
[
"MFR-L",
"KOLL",
52.93128,
6.962956,
1,
1.129,
true
]
[
"MFR-L",
"Takst",
52.9523773,
6.981644,
1.3,
1.809,
false
]
Here is another approach which uses jq variables and string interpolation:
.data as $data
| .stores[]
| "\($data),\(.name),\(.lat),\(.lng),\(.dist),\(.x10),\(.isOpen)"
output with sample data:
"MFR-L,KOLL,52.93128,6.962956,1,1.129,true"
"MFR-L,Takst,52.9523773,6.981644,1.3,1.809,false"
Given a JSON stream of the following form:
{ "a": 10, "b": 11 } { "a": 20, "b": 21 } { "a": 30, "b": 31 }
I would like to sum the values in each of the objects and output a single object, namely:
{ "a": 60, "b": 63 }
I'm guessing this will probably require flattening the above list of objects into a an array of [name, value] pairs and then summing the values using reduce but the documentation of the syntax for using reduce is woeful.
Unless your jq has inputs, you will have to slurp the objects up using the -s flag. Then you'll have to do a fair amount of manipulation:
Each of the objects needs to be mapped out to key/value pairs
Flatten the pairs to a single array
Group up the pairs by key
Map out each group accumulating the values to a single key/value pair
Map the pairs back to an object
map(to_entries)
| add
| group_by(.key)
| map({
key: .[0].key,
value: map(.value) | add
})
| from_entries
With jq 1.5, this could be greatly improved: You can do away with slurping and just read the inputs directly.
$ jq -n '
reduce (inputs | to_entries[]) as {$key,$value} ({}; .[$key] += $value)
' input.json
Since we're simply accumulating all the values in each of the objects, it'll be easier to just run through the key/value pairs of all the inputs, and add them all up.
I faced the same question when listing all artifacts from GitHub (see here for details) and want to sum their size.
curl https://api.github.com/repos/:owner/:repo/actions/artifacts \
-H "Accept: application/vnd.github.v3+json" \
-H "Authorization: token <your_pat_here>" \
| jq '.artifacts | map(.size_in_bytes) | add'
Input:
{
"total_count": 3,
"artifacts": [
{
"id": 0000001,
"node_id": "MDg6QXJ0aWZhY3QyNzUxNjI1",
"name": "artifact-1",
"size_in_bytes": 1,
"url": "https://api.github.com/repos/:owner/:repo/actions/artifacts/2751625",
"archive_download_url": "https://api.github.com/repos/:owner/:repo/actions/artifacts/2751625/zip",
"expired": false,
"created_at": "2020-03-10T18:21:23Z",
"updated_at": "2020-03-10T18:21:24Z"
},
{
"id": 0000002,
"node_id": "MDg6QXJ0aWZhY3QyNzUxNjI0",
"name": "artifact-2",
"size_in_bytes": 2,
"url": "https://api.github.com/repos/:owner/:repo/actions/artifacts/2751624",
"archive_download_url": "https://api.github.com/repos/:owner/:repo/actions/artifacts/2751624/zip",
"expired": false,
"created_at": "2020-03-10T18:21:23Z",
"updated_at": "2020-03-10T18:21:24Z"
},
{
"id": 0000003,
"node_id": "MDg6QXJ0aWZhY3QyNzI3NTk1",
"name": "artifact-3",
"size_in_bytes": 3,
"url": "https://api.github.com/repos/docker/mercury-ui/actions/artifacts/2727595",
"archive_download_url": "https://api.github.com/repos/:owner/:repo/actions/artifacts/2727595/zip",
"expired": false,
"created_at": "2020-03-10T08:46:08Z",
"updated_at": "2020-03-10T08:46:09Z"
}
]
}
Output:
6
Another approach, which illustrates the power of jq quite nicely, is to use a filter named "sum" defined as follows:
def sum(f): reduce .[] as $row (0; . + ($row|f) );
To solve the particular problem at hand, one could then use the -s (--slurp) option as mentioned above, together with the expression:
{"a": sum(.a), "b": sum(.b) } # (2)
The expression labeled (2) only computes the two specified sums, but it is easy to generalize, e.g. as follows:
# Produce an object with the same keys as the first object in the
# input array, but with values equal to the sum of the corresponding
# values in all the objects.
def sumByKey:
. as $in
| reduce (.[0] | keys)[] as $key
( {}; . + {($key): ($in | sum(.[$key]))})
;