jq select() and sort_by() - json

I am parsing a curl output from gitlab api, and I need to add a sort_by to my query, then select only certain values.
sample input:
[
{
"id": 10,
"name": "another-test",
"path": "another-test",
"description": "",
"visibility": "private",
"lfs_enabled": true,
"avatar_url": null,
"web_url": "https://mygitlab/groups/another-test",
"request_access_enabled": false,
"full_name": "another-test",
"full_path": "another-test",
"parent_id": 9
},
{
"id": 11,
"name": "asdfg",
"path": "asdfg",
"description": "",
"visibility": "private",
"lfs_enabled": true,
"avatar_url": null,
"web_url": "https://mygitlab/groups/asdfg",
"request_access_enabled": false,
"full_name": "asdfg",
"full_path": "asdfg",
"parent_id": 7
}
I parse the JSON with jq as follows:
curl http://..... | jq -r '.[] | select(.parent_id!=null) | .name, .parent_id'
This works exactly as expected, but when I try to sort the results by parent_id, I get an error:
curl http://..... | jq -r '.[] | select(.parent_id!=null) | .name, .parent_id | sort_by(.parent_id)'
jq: error (at <stdin>:0): Cannot index number with string "parent_id"
I can use sort_by(), by putting a single dot instead than .[]:
curl http://..... | jq '. | sort_by(.parent_id) '
But I cannot combine the 2 functions.
Clarification: I need to extract name and parent_id, sorted by parent_id, when it is not null.
Thanks in advance

jq's sort_by() function accepts an array as input.
curl 'http://...' |
jq -r '
map(select(.parent_id != null))
| sort_by(.parent_id)[]
| [.name, .parent_id]
| #tsv
'
Sample output:
asdfg 7
another-test 9

Related

jq - select objects and print null for missing

I'm trying to generate a CSV of sort from json file, the files are as below
cat role1.json
{
"Tags": [
{
"Key": "Name",
"Value": "Role1Name"
},
{
"Key": "ID",
"Value": "Role1ID"
},
{
"Key": "Manager",
"Value": "Role1Manager"
},
{
"Key": "User",
"Value": "Role1User"
},
{
"Key": "Country",
"Value": "USA"
}
]
}
cat role2.json
{
"Tags": [
{
"Key": "Name",
"Value": "Role2Name"
},
{
"Key": "ID",
"Value": "Role2ID"
},
{
"Key": "City",
"Value": "NewYork"
},
{
"Key": "Creator",
"Value": "Role2Creator"
},
{
"Key": "User",
"Value": "Role2User"
}
]
}
cat role3.json
{
"Tags": [
{
"Key": "Name",
"Value": "Role3Name"
},
{
"Key": "ID",
"Value": "Role3ID"
},
{
"Key": "Creator",
"Value": "Role3Creator"
},
{
"Key": "ZIP",
"Value": 82378
},
{
"Key": "Manager",
"Value": "Role3Manager"
},
{
"Key": "User",
"Value": "Role3User"
}
]
}
I want to generate lines from each of these to be later used as CSV, something like:
Role1Name, Role1ID, null, Role1Manager, Role1User
Role2Name, Role2ID, Role2Creator, null, Role2User
Role3Name, Role3ID, Role3Creator, Role3Manager, Role3User
For the header line
Name, ID, Creator, Manager, User
I'm able to get all the "Value" but not able to print null for missing "Key"
$cat role1.json | jq -rc '[.Tags[] | select(.Key == ("Name","ID","Creator","Manager","User")) | .Value]'
["Role1Name","Role1ID","Role1Manager","Role1User"]
$cat role2.json | jq -rc '[.Tags[] | select(.Key == ("Name","ID","Creator","Manager","User")) | .Value]'
["Role2Name","Role2ID","Role2Creator","Role2User"]
$cat role3.json | jq -rc '[.Tags[] | select(.Key == ("Name","ID","Creator","Manager","User")) | .Value]'
["Role3Name","Role3ID","Role3Creator","Role3Manager","Role3User"]
Can someone share with me how this can be done using jq.
Also, how can we enforce the order.
Thanks!
The key (ha!) is
[ .[ $keys[] ] ]
Had you looked at other answers to questions relating to CSV, you might have noticed the first step taken is to get the list of keys. This is often done by collecting the keys of the input objects. (Example) In your case, you have a hard-coded list, so it's even simpler.
If you wanted actual CSV, you could use
jq -sr '
[ "Name", "ID", "Creator", "Manager", "User" ] as $keys |
(
$keys,
( .[].Tags | from_entries | [ .[ $keys[] ] ] )
) |
#csv
' role*.json
This produces
"Name","ID","Creator","Manager","User"
"Role1Name","Role1ID",,"Role1Manager","Role1User"
"Role2Name","Role2ID","Role2Creator",,"Role2User"
"Role3Name","Role3ID","Role3Creator","Role3Manager","Role3User"
jqplay
Without a header:
jq -r '.Tags | from_entries | [ .["Name","ID","Creator","Manager","User"] ] | #csv' role*.json
jqplay
To get the specific output you posted (which isn't CSV), you could use
jq -sr '
[ "Name", "ID", "Creator", "Manager", "User" ] as $keys |
(
$keys,
( .[].Tags | from_entries | [ .[ $keys[] ] | . // "null" ] )
) |
join(", ")
' role*.json
This produces
Name, ID, Creator, Manager, User
Role1Name, Role1ID, null, Role1Manager, Role1User
Role2Name, Role2ID, Role2Creator, null, Role2User
Role3Name, Role3ID, Role3Creator, Role3Manager, Role3User
jqplay
Without a header:
jq -r '.Tags | from_entries | [ .["Name","ID","Creator","Manager","User"] | . // "null" ] | join(", ")' role*.json
jqplay
Got an answer from another forum, might be useful for others
$jq -rc '.Tags | from_entries | [.Name, .ID, .Creator, .Manager, .User]' role*.json
["Role1Name","Role1ID",null,"Role1Manager","Role1User"]
["Role2Name","Role2ID","Role2Creator",null,"Role2User"]
["Role3Name","Role3ID","Role3Creator","Role3Manager","Role3User"]

Including empty JSON values in jq output

I'm trying to get a .csv out that includes occasional empty values.
Calling this API (https://www.campaignmonitor.com/api/subscribers/#getting-subscribers-details) I get the following:
[
{
"ID": "fc0ce7105baeaf97f47c99be31d02a91",
"Type": "Campaign",
"Name": "Campaign One",
"Actions": [
{
"Event": "Open",
"Date": "2010-10-12 13:18:00",
"IPAddress": "192.168.126.87",
"Detail": ""
},
{
"Event": "Click",
"Date": "2010-10-12 13:16:00",
"IPAddress": "192.168.126.87",
"Detail": "https://example.com/post/12323/"
}
]
}
{
"ID": "dsadsamdkl9309ujd432",
"Type": "Campaign",
"Name": "Campaign Two",
"Actions": []
}
]
What I want to get as output:
"Campaign One","Open"
"Campaign One","Click"
"Campaign Two","none"
What I currently get
"Campaign One","Open"
"Campaign One","Click"
I can't seem to find a way to include values when "Actions" == []
What I tried so far:
Attempt 1:
curl -u "apikey:x" https://api.createsend.com/api/v3.2/subscribers/listID/history.json?email=example#email.com | jq -r '.[] | .Name as $n | .Actions[] | ([$n, .Event | if . == null then "none" else . end]) | #csv'
Attempt 2:
curl -u "apikey:x" https://api.createsend.com/api/v3.2/subscribers/listID/history.json?email=example#email.com | jq -r '.[] | .Name as $n | .Actions[] | ([$n, .Event // "none"]) | #csv'
Attempt 3:
curl -u "apikey:x" https://api.createsend.com/api/v3.2/subscribers/listID/history.json?email=example#email.com | jq -r '.[] | .Name as $n | .Actions[] |.Actions[] | if . == [] then .Actions[].Event = "" else . end | ([$n, .Event]) | #csv'
With the alternative operator //:
jq -r '.[] | (.Actions[].Event // "none") as $e | [ .Name, $e ] | #csv'
This assumes that the missing comma on line 20 hast been inserted.

JQ - Denormalize nested object

I've been trying to convert some JSON to csv and I have the following problem:
I have the following input json:
{"id": 100, "a": [{"t" : 1,"c" : 2 }, {"t": 2, "c" : 3 }] }
{"id": 200, "a": [{"t": 2, "c" : 3 }] }
{"id": 300, "a": [{"t": 1, "c" : 3 }] }
And I expect the following CSV output:
id,t1,t2
100,2,3
200,,3
300,3,
Unfortunately JQ doesn't output if one of select has no match.
Example:
echo '{ "id": 100, "a": [{"t" : 1,"c" : 2 }, {"t": 2, "c" : 3 }] }' | jq '{t1: (.a[] | select(.t==1)).c , t2: (.a[] | select(.t==2)).c }'
output:
{ "t1": 2, "t2": 3 }
but if one of the objects select returns no match it doesn't return at all.
Example:
echo '{ "id": 100, "a": [{"t" : 1,"c" : 2 }] }' | jq '{t1: (.a[] | select(.t==1)).c , t2: (.a[] | select(.t==2)).c }'
Expected output:
{ "t1": 2, "t2": null }
Does anyone know how to achieve this with JQ?
EDIT:
Based on a comment made by #peak I found the solution that I was looking for.
jq -r '["id","t1","t2"],[.id, (.a[] | select(.t==1)).c//null, (.a[] | select(.t==2)).c//null ]|#csv'
The alternative operator does exactly what I was looking for.
Alternative Operator
Here's a simple solution that does not assume anything about the ordering of the items in the .a array, and easily generalizes to arbitrarily many .t values:
# Convert an array of {t, c} to a dictionary:
def tod: map({(.t|tostring): .c}) | add;
["id", "t1", "t2"], # header
(inputs
| (.a | tod) as $dict
| [.id, (range(1;3) as $i | $dict[$i|tostring]) ])
| #csv
Command-line options
Use the -n option (because inputs is being used), and the -r option (to produce CSV).
This is an absolute mess, but it works:
$ cat tmp.json
{"id": 100, "a": [{"t" : 1,"c" : 2 }, {"t": 2, "c" : 3 }] }
{"id": 200, "a": [{"t": 2, "c" : 3 }] }
{"id": 300, "a": [{"t": 1, "c" : 3 }] }
$ cat filter.jq
def t(id):
.a |
map({key: "t\(.t)", value: .c}) |
({t1:null, t2:null, id:id} | to_entries) + . | from_entries
;
inputs |
map(.id as $id | t($id)) |
(.[0] | keys) as $hdr |
([$hdr] + map(to_entries |map(.value)))[]|
#csv
$ jq -rn --slurp -f filter.jq tmp.json
"id","t1","t2"
2,3,100
,3,200
3,,300
In short, you produce a direct object containing the values from your input, then add it to a "default" object to fill in the missing keys.

Converting 1-to-many json into csv

I'm trying to parse json output from an API call. The output has an array of orders, and each order has an array of items. I want to parse the output such that I have a single CSV output of each individual item with its parent order ID.
So if a single order contains multiple items, I need the orderID repeated for each item in its order. I've read the jq documentation and dozens of samples, and I've tried some trial and error for hours. I'm SO confused as to how to do this.
I'm struggling very much with the jq parsing syntax. None of the examples are really helping, and I'm just confused. Here's the basics:
curl -s https://api.site.com/orders?page=1&pageSize=10 | jq '.'
A sample of the json is below.
{
"orders": [
{
"orderId": 217356098,
"items": [
{
"orderItemId": 327010821,
"lineItemKey": "1",
"sku": "AJC-C10S",
"name": "TestDescription",
"imageUrl": null,
"weight": null,
"quantity": 2,
"unitPrice": 106.85,
"taxAmount": null,
"shippingAmount": null,
"warehouseLocation": null,
"options": [],
"productId": null,
"fulfillmentSku": null,
"adjustment": false,
"upc": null,
"createDate": "2016-11-09T02:11:28.307",
"modifyDate": "2016-11-09T02:11:28.307"
},
{
"orderItemId": 327010822,
"lineItemKey": "1",
"sku": "AJC-C106",
"name": "AnotherTestDescription",
"imageUrl": null,
"weight": null,
"quantity": 2,
"unitPrice": 106.85,
"taxAmount": null,
"shippingAmount": null,
"warehouseLocation": null,
"options": [],
"productId": null,
"fulfillmentSku": null,
"adjustment": false,
"upc": null,
"createDate": "2016-11-09T02:11:28.307",
"modifyDate": "2016-11-09T02:11:28.307"
}
]
},
],
"total": 359934,
"page": 1,
"pages": 179968
}
Expected output (without column headers of course):
orderId,orderItemId,sku,name
217356098,327010821,"JC-C10S","TestDescription"
217356098,327010822,"JC-C106","AnotherTestDescription"
As you can see, each item has its own line, but if they came from the same order, the orderId should be repeated on each line.
How can I do this?
With the -r command-line option, the following jq filter:
.orders[]
| .orderId as $oid
| .items[]
| [$oid, .orderItemId, .sku, .name]
| #csv
produces the desired output.
If there's any chance that any of the selected values might be [], then consider adding a line like the following immediately before the last line above:
| map_values(if . == [] then "NONE" else . end)
Thanks! That worked with a slight alteration:
.orders[]
| .orderId as $oid
| .items[]
| [$oid, .items.orderItemId, .items.sku, .items.name | tostring]
| #csv

How to create key:value list from JSON? Key name should contain some values from object itself

I'm trying to parse JSON and store certain values as metrics in Graphite.
In order to make my Graphite more user-friendly I have to form a metric name, that contains some values from its object.
I got working solution on bash loops + jq, but it's really slow. So I'm asking for help :)
Here is my input:
{
...
},
"Johnny Cage": {
"firstname": "Johnny",
"lastname": "Cage",
"height": 183,
"weight": 82,
"hands": 2,
"legs": 2,
...
},
...
}
Desired output:
mk.fighter.Johnny.Cage.firstname Johnny
mk.fighter.Johnny.Cage.lastname Cage
mk.fighter.Johnny.Cage.height 183
mk.fighter.Johnny.Cage.weight 82
mk.fighter.Johnny.Cage.hands 2
mk.fighter.Johnny.Cage.legs 2
...
With single jq command:
Sample input.json:
{
"Johnny Cage": {
"firstname": "Johnny",
"lastname": "Cage",
"height": 183,
"weight": 82,
"hands": 2,
"legs": 2
}
}
jq -r 'to_entries[] | (.key | sub(" "; ".")) as $name
| .value | to_entries[]
| "mk.fighter.\($name).\(.key) \(.value)"' input.json
To get $name as a combination of inner firstname and lastname keys replace (.key | sub(" "; ".")) as $name with "\(.value.firstname).\(.value.lastname)" as $name
The output:
mk.fighter.Johnny.Cage.firstname Johnny
mk.fighter.Johnny.Cage.lastname Cage
mk.fighter.Johnny.Cage.height 183
mk.fighter.Johnny.Cage.weight 82
mk.fighter.Johnny.Cage.hands 2
mk.fighter.Johnny.Cage.legs 2