Sort descending by multiple keys in jq - json

I have the following array:
[{
"name": "Object 1",
"prop1": 5,
"prop2": 2
}, {
"name": "Object 2",
"prop1": 6,
"prop2": 4
}, {
"name": "Object 3",
"prop1": 5,
"prop2": 3
}]
I want to sort this array analogous to this SQL ORDER BY prop1 DESC, prop2 ASC so I have this result:
[{
"name": "Object 2",
"prop1": 6,
"prop2": 4
}, {
"name": "Object 1",
"prop1": 5,
"prop2": 2
}, {
"name": "Object 3",
"prop1": 5,
"prop2": 3
}]
How can I sort an array a) descending by a key and b) by multiple keys?
Version: jq 1.5

In jq, arrays sort by the sorting of the elements they contain, in order. That is:
$ jq -n '[1, 2] < [1, 3], [1, 2] < [2, 1]'
true
true
The sort_by filter sorts an array, taking an expression as argument that will be evaluated for each member of the array. For example, if you wanted to sort a list of words by length:
$ jq -n '["prop", "leo", "column", "blast"] | sort_by(length)'
[
"leo",
"prop",
"blast",
"column"
]
If the expression given to sort_by as argument returns more than one value, the return values will be implicitly wrapped in an array, which will be subject to the array sorting rules referred to above. For example, if you wanted to sort a list of words by length, and then alphabetically:
$ jq -n '["pro", "leo", "column", "ablast"] | sort_by(length, .)'
[
"leo",
"pro",
"ablast",
"column"
]
Knowing this, and taking into account that the values in your example are numeric, you can just do the following:
$ jq 'sort_by(-.prop1, .prop2)'

One may use the reverse (jq 1.5 Manual) function like:
$ jq -n '["pro", "leo", "column", "ablast"] | sort | reverse'
[
"pro",
"leo",
"column",
"ablast"
]
So your specific example may become:
$ jq -n '[{...}, {...}, {...}, {...}] | sort_by(.prop2) | reverse | sort_by(.prop1) | reverse'
[
{
"name": "Object 2",
"prop1": 6,
"prop2": 4
},
{
"name": "Object 1",
"prop1": 5,
"prop2": 2
},
{
"name": "Object 3",
"prop1": 5,
"prop2": 3
}
]
SQL ORDER BY prop1 DESC, prop2 ASC → jq | sort_by(.prop2) | reverse | sort_by(.prop1) | reverse – Note, the sorting is specified with properties in the reverse order, and reverse is used twice.
Given prop1 and prop2 are numeric, the accepted answer (sort_by(-.prop1, .prop2)) is much simpler/better.

Related

how to format the result with jq

I have the following printout,
{
"metric": {
"container": "container1",
"namespace": "namespace1",
"pod": "pod1"
},
"values": [
[
1664418600,
"1"
],
[
1664418900,
"2"
],
[
1664419200,
"6"
],
[
1664419500,
"8"
],
[
1664419800,
"7"
],
[
1664420100,
"9"
]
]
}
{
"metric": {
"container": "container2",
"namespace": "namespace2",
"pod": "pod2"
},
"values": [
[
1664420100,
"1"
]
]
}
What I want:
container=container1,namespace=namespace1,pod=pod1
1 1664418600
2 1664418900
6 1664419200
8 1664419500
7 1664419800
9 1664420100
container=container2,namespace=namespace2,pod=pod2
1 1664420100
Build it from two JSON programs:
Header lines: .metric | to_entries | map(join("=")) | join(",")
Get metric object: .metric
Convert to an array of key-value pairs: to_entries
Map each key-value pair object to a string "key=value": map(join("="))
Join all pairs by comma: join(",")
Value lists: .values[] | [last,first] | join(" ")
Stream values: .values[]
Reverse each two-valued array: [last,first]
Join items by blank: join(" ")
An alternative for 2.2. and 2.3. could be "\(last) \(first)", i.e. values[] | "\(last) \(first)". Or [last,first] could be replaced with reverse: .values[] | reverse | join(" ").
Putting the two programs together:
(.metric | to_entries | map(join("=")) | join(",")),
(.values[] | [last,first] | join(" "))
And then execute with raw output enabled: jq -r (.metrics|to_entries…
Output:
container=container1,namespace=namespace1,pod=pod1
1 1664418600
2 1664418900
6 1664419200
8 1664419500
7 1664419800
9 1664420100
container=container2,namespace=namespace2,pod=pod2
1 1664420100

how to extract and modify inner array objects with parent object data in jq

We are tying to format a json similar to this:
[
{"id": 1,
"type": "A",
"changes": [
{"id": 12},
{"id": 13}
],
"wanted_key": "good",
"unwanted_key": "aaa"
},
{"id": 2,
"type": "A",
"unwanted_key": "aaa"
},
{"id": 3,
"type": "B",
"changes": [
{"id": 31},
{"id": 32}
],
"unwanted_key": "aaa",
"unwanted_key2": "aaa"
},
{"id": 4,
"type": "B",
"unwanted_key3": "aaa"
},
null,
null,
{"id": 7}
]
into something like this:
[
{
"id": 1,
"type": "A",
"wanted_key": true # every record must have this key/value
},
{
"id": 12, # note: this was in the "changes" property of record id 1
"type": "A", # type should be the same type than record id 1
"wanted_key": true
},
{
"id": 13, # note: this was in the "changes" property of record id 1
"type": "A", # type should be the same type than record id 1
"wanted_key": true
},
{
"id": 2,
"type": "A",
"wanted_key": true
},
{
"id": 3,
"type": "B",
"wanted_key": true
},
{
"id": 31, # note: this was in the "changes" property of record id 3
"type": "B", # type should be the same type than record id 3
"wanted_key": true
},
{
"id": 32, # note: this was in the "changes" property of record id 3
"type": "B", # type should be the same type than record id 3
"wanted_key": true
},
{
"id": 4,
"type": "B",
"wanted_key": true
},
{
"id": 7,
"type": "UNKN", # records without a type should have this type
"wanted_key": true
}
]
So far, I've been able to:
remove null records
obtain the keys we need with their default
give records without a type a default type
What we are missing:
from records having a changes key, create new records with the type of their parent record
join all records in a single array
Unfortunately we are not entirely sure how to proceed... Any help would be appreciated.
So far our jq goes like this:
del(..|nulls) | map({id, type: (.type // "UNKN"), wanted_key: (true)}) | del(..|nulls)
Here's our test code:
https://jqplay.org/s/eLAWwP1ha8P
The following should work:
map(select(values))
| map(., .type as $type | (.changes[]? + {$type}))
| map({id, type: (.type // "UNKN"), wanted_key: true})
Only select non-null values
Return the original items followed by their inner changes array (+ outer type)
Extract 3 properties for output
Multiple map calls can usually be combined, so this becomes:
map(
select(values)
| ., (.type as $type | (.changes[]? + {$type}))
| {id, type: (.type // "UNKN"), wanted_key: true}
)
Another option without variables:
map(
select(values)
| ., .changes[]? + {type}
| {id, type: (.type // "UNKN"), wanted_key: true}
)
# or:
map(select(values))
| map(., .changes[]? + {type})
| map({id, type: (.type // "UNKN"), wanted_key: true})
or even with a separate normalization step for the unknown type:
map(select(values))
| map(.type //= "UNKN")
| map(., .changes[]? + {type})
| map({id, type, wanted_key: true})
# condensed to a single line:
map(select(values) | .type //= "UNKN" | ., .changes[]? + {type} | {id, type, wanted_key: true})
Explanation:
Select only non-null values from the array
If type is not set, create the property with value "UNKN"
Produce the original array items, followed by their nested changes elements extended with the parent type
Reshape objects to only contain properties id, type, and wanted_key.
Here's one way:
map(
select(values)
| (.type // "UNKN") as $type
| ., .changes[]?
| {id, $type, wanted_key: true}
)
[
{
"id": 1,
"type": "A",
"wanted_key": true
},
{
"id": 12,
"type": "A",
"wanted_key": true
},
{
"id": 13,
"type": "A",
"wanted_key": true
},
{
"id": 2,
"type": "A",
"wanted_key": true
},
{
"id": 3,
"type": "B",
"wanted_key": true
},
{
"id": 31,
"type": "B",
"wanted_key": true
},
{
"id": 32,
"type": "B",
"wanted_key": true
},
{
"id": 4,
"type": "B",
"wanted_key": true
},
{
"id": 7,
"type": "UNKN",
"wanted_key": true
}
]
Demo
Something like below should work
map(
select(type == "object") |
( {id}, {id : ( .changes[]? .id )} ) +
{ type: (.type // "UNKN"), wanted_key: true }
)
jq play - demo

group objects by a field and sum another, then produce a CSV report

How can I create a csv from this json? I have:
[
{
"name": "John",
"cash": 5
},
{
"name": "Anna",
"cash": 4
},
{
"name": "Anna",
"cash": 3
},
{
"name": "John",
"cash": 8
}
]
I need group by name and sum the cash and send the result a .csv like:
John,13
Anna,7
Thanks!
JQ has group_by as a builtin, use that and do map(.cash) | add to sum cash values for each group.
group_by(.name)[] | [.[0].name, (map(.cash) | add)] | #csv
Online demo

jq select() and sort_by()

I am parsing a curl output from gitlab api, and I need to add a sort_by to my query, then select only certain values.
sample input:
[
{
"id": 10,
"name": "another-test",
"path": "another-test",
"description": "",
"visibility": "private",
"lfs_enabled": true,
"avatar_url": null,
"web_url": "https://mygitlab/groups/another-test",
"request_access_enabled": false,
"full_name": "another-test",
"full_path": "another-test",
"parent_id": 9
},
{
"id": 11,
"name": "asdfg",
"path": "asdfg",
"description": "",
"visibility": "private",
"lfs_enabled": true,
"avatar_url": null,
"web_url": "https://mygitlab/groups/asdfg",
"request_access_enabled": false,
"full_name": "asdfg",
"full_path": "asdfg",
"parent_id": 7
}
I parse the JSON with jq as follows:
curl http://..... | jq -r '.[] | select(.parent_id!=null) | .name, .parent_id'
This works exactly as expected, but when I try to sort the results by parent_id, I get an error:
curl http://..... | jq -r '.[] | select(.parent_id!=null) | .name, .parent_id | sort_by(.parent_id)'
jq: error (at <stdin>:0): Cannot index number with string "parent_id"
I can use sort_by(), by putting a single dot instead than .[]:
curl http://..... | jq '. | sort_by(.parent_id) '
But I cannot combine the 2 functions.
Clarification: I need to extract name and parent_id, sorted by parent_id, when it is not null.
Thanks in advance
jq's sort_by() function accepts an array as input.
curl 'http://...' |
jq -r '
map(select(.parent_id != null))
| sort_by(.parent_id)[]
| [.name, .parent_id]
| #tsv
'
Sample output:
asdfg 7
another-test 9

Parse JSON Nested SubValues in Powershell to Table

I converted the JSON string to Powershell in v5. The original json string is below:
$j = #'
[{
"id": "1",
"Members": [
"A",
"B",
"C"
]
}, {
"id": "2",
"Members": [
"A",
"C"
]
}, {
"id": "3",
"Members": [
"A",
"D"
]
}]
'#
$json = $j | ConvertFrom-Json
I would like the result set to look like the result set below. Eventually I will export to SQL:
id Members
----- --------
1 A
1 B
1 C
2 A
2 C
3 A
3 D
try this
$json | % {
$id = $_.id
$_.members | select #{n='id';e={$id}}, #{n='members';e={$_}}
}