Given an array of JSON objects, I'd like to output a CSV where one of the rows contains each object key and the others are based on each object value.
The input json is:
{
"PCID000": {
"OSmodle": "LINUX",
"IEversion": "2.15.0",
"hardwareUSB": [
"Card reader",
"keyboard"
],
"OrderStatus": "01"
},
"PCID999": {
"OSmodle": "LINUX",
"OSversion": "4.0",
"hardwareUSB": [],
"OrderStatus": "01"
}
}
The output would look something like this. The header can be hardcoded.
PCID,OSmodle,OSversion,IEversion,hardwareUSB, OrderStatus
"PCID000","LINUX",,"2.15.0","Card reader&keyboard","01"
"PCID999","LINUX","4.0",,"01
You can use the to_entries function to convert an object such as {"a": 1, "b": 2} to an array of key-value objects such as [{"key": "a", "value": 1}, {"key": "b", "value": 2}]. Then map over this to pick the key and the parts of the value of interest.
The jq script would look like this:
to_entries | map([
.key,
.value.OSmodle,
.value.OSversion,
.value.IEversion,
(.value.hardwareUSB | join("&")),
.value.OrderStatus])
| ["PCID", "OSmodle", "OSversion", "IEversion", "hardwareUSB", "OrderStatus"], .[]
| #csv
Output (with -r):
"PCID","OSmodle","OSversion","IEversion","hardwareUSB","OrderStatus"
"PCID000","LINUX",,"2.15.0","Card reader&keyboard","01"
"PCID999","LINUX","4.0",,"","01"
jqplay
Related
Lets say I have an I/P json file as below. And I want to extract the O/P in a CSV format with the below fields. Specifically, I want to get the value of the key "Gamma" in the o/p if the key "Gamma" exists in "tags" map. If the key doesn't exists, it should just print a NULL value. The expected o/p is below.
generated_time,platform,id,,
2021-09-09:12:03:12,earth,2eeee67748,Ray,2021-08-25 09:41:06
2021-09-09:12:03:12,sun,xxxxx12334,NULL,2021-08-25 10:11:31
[
{
"generated_time": "generated_time",
"platform": "platform",
"id": "id"
},
{
"generated_time": "2021-09-09:12:03:12",
"platform": "earth",
"id": "2eeee67748",
"tags": {
"app": "map",
"Gamma": "Ray",
"null": [
"allow-all-humans"
]
},
"created": "2021-08-25 09:41:06"
},
{
"generated_time": "2021-09-09:12:03:12",
"platform": "sun",
"id": "xxxxx12334",
"tags": {
"component": "machine",
"environment": "hot",
"null": [
"aallow-all-humans"
]
},
"created": "2021-08-25 10:11:31"
}
]
jq has a builtin #csv which renders an array
as CSV with double quotes for strings, and quotes escaped by repetition.
If the additional quoting (as compared to your expected output) isn't an issue, the following
jq --raw-output '
# produce an array for each element in the input array
.[] | [
# containing the first three columns unchanged
.generated_time, .platform, .id,
# if the input element has a field named "tags"
if has("tags")
# then add two more columns and replace an inexistant Gamma with "NULL"
then (.tags.Gamma // "NULL", .created)
# otherwise add two empty columns instead
else (null, null) end
# and convert the array into CSV format
] | #csv
' input.json
will produce
"generated_time","platform","id",,
"2021-09-09:12:03:12","earth","2eeee67748","Ray","2021-08-25 09:41:06"
"2021-09-09:12:03:12","sun","xxxxx12334","NULL","2021-08-25 10:11:31"
I have a process that generates a JSON object containing some "header" values as scalars and a number of payload values as arrays:
{
"header 1": 42,
"header 2": "2020-01-27",
"payload 1": [
{
"foo": 1
},
{
"foo": 2
}
],
"another payload": [
10,
9,
8,
7
]
}
I have been able to isolate the names of the array fields with the following command:
$ jq '[to_entries | .[] | select(.value | type == "array")] | from_entries | keys_unsorted' results.json
[
"payload 1",
"another payload"
]
But I don't know how to use this to get the lengths of the arrays. The output I'm looking for would be something like:
{
"payload 1": 2,
"another payload": 4
}
Or anything that lists the keys of fields that are arrays and the length of the arrays.
What is a jq command to list the lengths of all array fields in the top-level object?
You don't need *_entries functions here.
map_values(arrays | length)
Online demo
I use jq 1.6 in a Windows 10 PowerShell enviroment and trying to select keys from coincidentally numeric json objects.
Json exampel:
{
"alliances_info":{
"744085325458334213":{
"emblem":3,
"name":"wellwell",
"member_count":1,
"level":1,
"military_might":1035,
"public":false,
"tag":"MELL",
"slogan":"",
"id":744085325458334213
},
"744128593839677958":{
"emblem":0,
"name":"Brave",
"member_count":1,
"level":1,
"military_might":1035,
"public":false,
"tag":"GABA",
"slogan":"",
"id":744128593839677958
},
"746034084459209223":{
"emblem":0,
"name":"Queen",
"member_count":1,
"level":1,
"military_might":1035,
"public":false,
"tag":"QUE",
"slogan":"",
"id":746034084459209223
},
"750446471312466445":{
"emblem":0,
"name":"Phoenix Inc",
"member_count":35,
"level":6,
"military_might":453369,
"public":true,
"tag":"PHOI",
"slogan":"",
"id":750446471312466445
},
"750446518934594062":{
"emblem":11,
"name":"Australia",
"member_count":44,
"level":8,
"military_might":957211,
"public":true,
"tag":"AUST",
"slogan":"Go Australia",
"id":750446518934594062
}
},
"server_version":"v7.190.4-master.000000006"
}
I tried several jq commands:
.alliances_info | .[] | [{alliance_name: .name, alliance_count: .member_count, alliance_level: .level, alliance_power: .military_might, alliance_tag: .tag, alliance_slogan: .slogan, alliance_id: .id}]
or
.alliances_info | .. | objects | [{alliance_name: .name, alliance_c
ount: .member_count, alliance_level: .level, alliance_power: .military_might, alliance_tag: .tag, alliance_slogan: .slog
an, alliance_id: .id}]
But Always get a jq error: parse error: Invalid numeric literal at line 1, column 3
I renounce on the object Building in the first command (and built only a Array) it works. But i need that objects. Any tips?
BR
Timo
Your first query works perfectly well with the given JSON sample. Perhaps you're invoking jq incorrectly. If you have the jq program in a file, say select.jq, you'd invoke jq like so:
jq -f select.jq sample.json
If that doesn't help, then try:
jq empty sample.json
If that fails, there might be something wrong with the encoding of the JSON.
I'm not sure I understand what you want.
Your first attempt works for me, but generates one output for JSON value in the input. That is, I created a file named so.json and put in it your JSON from above:
{
"alliances_info": {
"744085325458334213": {
"emblem": 3,
⋮
}
When I run your program , I get:
$ jq '.alliances_info | .[] | [{alliance_name: .name, alliance_count: .member_count, alliance_level: .level, alliance_power: .military_might, alliance_tag: .tag, alliance_slogan: .slogan, alliance_id: .id}]' so.json
[
{
"alliance_name": "wellwell",
"alliance_count": 1,
"alliance_level": 1,
"alliance_power": 1035,
"alliance_tag": "MELL",
"alliance_slogan": "",
"alliance_id": 744085325458334200
}
]
[
{
"alliance_name": "Brave",
⋮
]
If you want an array at all, you probably want one array containing all the alliances like this:
$ jq '.alliances_info | [ .[] | { alliance_name: .name, alliance_id: .id } ]' so.json
[
{
"alliance_name": "wellwell",
"alliance_id": 744085325458334200
},
{
"alliance_name": "Brave",
"alliance_id": 744128593839678000
},
{
"alliance_name": "Queen",
"alliance_id": 746034084459209200
},
{
"alliance_name": "Phoenix Inc",
"alliance_id": 750446471312466400
},
{
"alliance_name": "Australia",
"alliance_id": 750446518934594000
}
]
Starting from the left,
- .alliances_info looks in its input object for the field named "alliances_info" and outputs its value
- the | next says take the output from the left-hand side and pass those as inputs to the right-hand side.
- right after that first |, I have a [ «jq expressions» ] which tells jq to create one JSON array output for each input; the elements of that array are the outputs of that inner «jq expressions»
- that inner expression starts with .[] which means to produce one output for each JSON value (ignoring the keys) in the input object. For us, that will be the objects named "744085325458334213", "744128593839677958", …
- The next | uses those objects as input and for each, generates a JSON object { alliance_name: .name, alliance_id: .id }
That's why I end up with one JSON array containing 5 JSON objects.
As far as I can tell, you are mostly just renaming a bunch of the fields. For that, you could just do something like this:
$ jq --argjson renameMap '{ "name": "alliance_name", "member_count": "alliance_count", "level": "alliance_level", "military_might": "alliance_power", "tag": "alliance_tag", "slog": "alliance_slogan"}' '.alliances_info |= ( . | [ to_entries[] | ( .value |= ( . | [ to_entries[] | ( .key |= ( if $renameMap[.] then $renameMap[.] else . end ) ) ] | from_entries ) ) ] | from_entries )' so.json
{
"alliances_info": {
"744085325458334213": {
"emblem": 3,
"alliance_name": "wellwell",
"alliance_count": 1,
"alliance_level": 1,
"alliance_power": 1035,
"public": false,
"alliance_tag": "MELL",
"slogan": "",
"id": 744085325458334200
},
"744128593839677958": {
"emblem": 0,
"alliance_name": "Brave",
"alliance_count": 1,
"alliance_level": 1,
"alliance_power": 1035,
"public": false,
"alliance_tag": "GABA",
"slogan": "",
"id": 744128593839678000
},
⋮
},
"server_version": "v7.190.4-master.000000006"
}
well i am a idiot (to be here totally clear). I found the reason (and this is normally a nobrainer...). I read the input from a file and the funny thing is that the file is Unicode but no UTF8. after recoding the command is working fine. Thanks for the help.
BR
Timo
I have multiple json objects which could be less when i merge the arrays if a object key matches the same value as the next json object. I'm trying to accomplish this with jq.
I think i have to use group_by(.name) first to group matching keys. I'm also using slurp to first wrap all objects into one big array.
I don't have anything working for now.
given:
{
"name": "a",
"list": [ "a1", "a2" ]
}
{
"name": "a",
"list": [ "a3", "a4" ]
}
{
"name": "b",
"list": [ "b1", "b2" ]
}
should result in:
{
"name": "a",
"list": [ "a1", "a2", "a3", "a4" ]
}
{
"name": "b",
"list": [ "b1", "b2" ]
}
You can use reduce like this:
$ jq -c -n 'reduce inputs as $p ({}; .[$p.name] |= { name : $p.name, list : (.list + $p.list) }) | .[]' file
{"name":"a","list":["a1","a2","a3","a4"]}
{"name":"b","list":["b1","b2"]}
Here's a simple and efficient solution that uses a common "aggregate by" technique:
reduce inputs as $kv ({}; .[$kv.name] += $kv.list)
| keys_unsorted[] as $k
| {name: $k, list: .[$k]}
Since inputs has been used here, the -n command-line option of jq should be specified.
I'm trying to flatten a JSON consisting of nested objects. The top layer contains several key/value pairs, where each value is itself an array of a number of objects (the bottom layer).
What I would like to get, using jq, is simply an array of objects containing all the objects of the bottom layer, each of which with an additional key/value pair identifying the top-layer key it originally belonged to.
In other words, I would like to turn a JSON
{
"key1": [obj1, obj2],
"key2": [obj3]
}
into a plain array
[OBJ1, OBJ2, OBJ3]
where each OBJi is simply the original object with an extra key/value pair
"parent-key-name": keyx
where keyx would be the top-layer key obji belonged to, i.e. "key1" for obj1 and obj2, and "key2" for obj3.
I'm struggling with the fact that when referencing the objects in the bottom layer, e.g. via .[], jq does not seem to have inbuilt functionality to access associated top-layer information. However, I'm new to jq, and hope there is an easy solution after all.
Given the following input :
{
"key1": [{"name":"Emma"},{"name":"Bob"}],
"key2": [{"name":"Jean"}]
}
You can divide your items to entries, store the key in a variable and add the value for each item in value object:
jq '[ to_entries[] | .key as $parent | .value[] |
.["parent-key-name"] |= (.+ $parent) ] ' test.json
which gives the following output :
[
{
"name": "Emma",
"parent-key-name": "key1"
},
{
"name": "Bob",
"parent-key-name": "key1"
},
{
"name": "Jean",
"parent-key-name": "key2"
}
]
The solution presented below consists of two steps, each of which might be helpful separately, e.g. if someone wants to "flatten" the JSON in a slightly different way.
First, let's make the changes to obj[i] "in-place":
with_entries( .key as $k | .value[] |= ( . + {"parent-key-name": $k} ) )
Example:
$ jq -n -c -f program.jq
Input:
{
"key1": [{a:1}, {a:2}],
"key2": [{b:3}]
}
Output:
{
"key1": [
{
"a": 1,
"parent-key-name": "key1"
},
{
"a": 2,
"parent-key-name": "key1"
}
],
"key2": [
{
"b": 3,
"parent-key-name": "key2"
}
]
}
To flatten, simply append | [.[]] to the above filter. This produces:
[[{"a":1,"parent-key-name":"key1"},{"a":2,"parent-key-name":"key1"}],[{"b":3,"parent-key-name":"key2"}]]