CentOS, jq
https://stedolan.github.io/jq/manual/
I want to export json to csv.
I use tool jq for this.
Here example of json.
{
"page": {
"id": "kctbh9vrtdwd",
"name": "GitHub",
"url": "https://www.githubstatus.com",
"time_zone": "Etc/UTC",
"updated_at": "2021-05-27T16:56:02.461Z"
},
"status": {
"indicator": "none",
"description": "All Systems Operational"
}
}
I get by
curl -s https://www.githubstatus.com/api/v2/status.json
Here convert json to csv.
curl -s https://www.githubstatus.com/api/v2/status.json | jq -r '.page | [.id, .name] | #csv'
And here is the result:
"kctbh9vrtdwd","GitHub"
But why not print csv headers?
There is quite a lot of noise on the SO page that is provided as a link in one of the comments,
so here are two safe jq-only solutions ("safe" in the sense that it does not matter how the keys are ordered in the input JSON):
Manually add the headers
["id", "name"],
(.page | [.id, .name])
| #csv
Include the headers based on the specification of the relevant columns
["id", "name"] as $headers
| $headers, (.page | [.[$headers[]]])
| #csv
Related
In the below json, I'm unable to get the value which have reporter only.
the output should be jhoncena only which should written into a file.
jq -r '.values' response.json | grep reporter
the output for this is
"name": "reporter-jhoncena"
{
"size": 3,
"limit": 25,
"isLastPage": true,
"values": [
{
"name": "hello-world"
},
{
"name": "test-frame"
},
{
"name": "reporter-jhoncena"
}
],
"start": 0
}
You can use capture :
jq -r '.values[].name
| capture("^reporter-(?<name>.*)").name
' response.json
You can use split such as
jq -r '.values[2].name | split("-")[1]' response.json
Demo
Edit : Alternatively you can use
jq -r '.values[].name | select(.|split("-")[0]=="reporter")|split("-")[1]' response.json > outfile.txt
without knowing the order of the name element within the array
Demo
jq -r '.values[]
| select(.name|index("reporter"))
| .name
| sub("reporter-";"")' in.json > out.txt
Of course you might wish to use a different selection criterion, e.g. using startswith or test.
I am trying to convert JSON to CSV for selected keys using jq.
file.json
{
"_ref": "ipv4address/Li5pcHY0X2FkZHJlc3yMDIuMS8w:10.202.202.1",
"discovered_data": {
"bgp_as": 64638,
"device_model": "catalyst37xxStack",
"device_port_name": "Vl2002",
"device_port_type": "propVirtual",
"device_type": "Switch-Router",
"device_vendor": "Cisco",
"discovered_name": "Test_Device.network.local",
"discoverer": "Network Insight",
"first_discovered": 1580161888,
"last_discovered": 1630773758,
"mac_address": "aa:bb:cc:dd:ee:ff",
"mgmt_ip_address": "10.202.202.1",
"os": "15.2(4)E10",
"port_speed": "Unknown",
"port_vlan_name": "TEST-DATA",
"port_vlan_number": 2002
},
"ip_address": "10.202.202.1",
"is_conflict": false,
"mac_address": "",
"names": ["Test_Device"],
"network": "10.202.202.0/23",
"network_view": "TEST VIEW",
"objects": [],
"status": "USED",
"types": [
"UNMANAGED"
],
"usage": []
}
my desired output is:
names,ip_address,discovered_data.mac_address,discovered_data.discovered_name
Test_Device,10.202.202.1,aa:bb:cc:dd:ee:ff,Test_Device.network.local
So far, I have tried using following command but getting some syntax error:
jq -r 'map({names,ip_address,discovered_data.mac_address,discovered_data.discovered_name}) | (first | keys_unsorted) as $keys | map([to_entries[] | .value]) as $rows | $keys,$rows[] | #csv' < file.json
Assuming the JSON has been fixed, consider the output of:
(null
| {names,
ip_address,
"discovered_data.mac_address",
"discovered_data.discovered_name"} | keys_unsorted) as $keys
| $keys,
({names: .names[],
ip_address,
"discovered_data.mac_address": .discovered_data.mac_address,
"discovered_data.discovered_name": .discovered_data.discovered_name }
| [.[]])
| #csv
Assuming jq is invoked with the -r command-line option, this has the advantage of producing valid CSV. If you prefer to have all the key names and values unquoted, you might wish to consider using join(",") instead of #csv, or some more sophisticated variation if you want to have your cake and eat it.
I have 1000 odd json files. I want to convert them into single csv file. I want only specific attributes in the csv file.
Here is sample json file
{
"id": "90",
"productName": "XYZ",
"businessUnit": "",
"daemon": "MSU",
"features": [
{
"name": "MNQ",
"count": "10",
"id": "4"
},
{
"name": "PQR",
"count": "20",
"id": "5"
}
]
}
This is the sample output
ProductName,FeatureName, FeatureCount
XYZ,MNQ,10
XYZ,PQR,20
I want to extract following attributes "productName": "XYZ " and from "features" array. I want to extract "name" and "count" attributes to a CSV file.
I want to this to 1000 odd JSON files and write it to CSV file. How to do this?
Using Miller (https://github.com/johnkerl/miller) and running
mlr --j2c cut -r -f "(pro|name|count)" then reshape -r "feature" -o k,v then nest --explode --values --across-fields --nested-fs ":" -f k then reshape -s k_3,v then cut -x -r -f "_" input.json >output.csv
you have
+-------------+------+-------+
| productName | name | count |
+-------------+------+-------+
| XYZ | MNQ | 10 |
| XYZ | PQR | 20 |
+-------------+------+-------+
It's a commandline utility you can use in nearby every OS. Then you can do a simply for loop and merge all.
I have a JSON file from the Spotify API that lists all the songs on a specific album. The file is organized as follows:
.
.name
.tracks.items
.tracks.items[]
.tracks.items[].artists
.tracks.items[].artists[].name
.tracks.items[].duration_ms
.tracks.items[].name
I'm using jq to create a csv with the following information: song's artist, song's title, and song's duration. I can do this using the following syntax:
jq -r '.tracks.items[] | [.artists[].name, .name, .duration_ms] | #csv' myfile.json
Output:
"Michael Jackson","Wanna Be Startin' Somethin'",363400
"Michael Jackson","Baby Be Mine",260666
...
However, I would like to also add the value under .name (which represents the name of the album the songs are from) to every row of my csv file. Something that would look like this:
"Thriller","Michael Jackson","Wanna Be Startin' Somethin'",363400
"Thriller","Michael Jackson","Baby Be Mine",260666
...
Is it possible to do this using the #csv filter? I can do it by hand by hardcoding the name of the album like this
jq -r '.tracks.items[] | ["Thriller", .artists[].name, .name, .duration_ms] | #csv' myfile.json
But I was hoping there might be a nicer way to do it.
EDIT:
Here's what the file looks like:
{
"name": "Thriller",
"tracks": {
"items": [
{
"artists": [
{
"name": "Michael Jackson"
}
],
"duration_ms": 363400,
"name": "Wanna Be Startin' Somethin'"
},
{
"artists": [
{
"name": "Michael Jackson"
}
],
"duration_ms": 260666,
"name": "Baby Be Mine"
}
]
}
}
See the "Variable / Symbolic Binding Operator" section in jq's documentation
jq -r '
.name as $album_name ### <- THIS RIGHT HERE
| .tracks.items[]
| [$album_name, .artists[].name, .name, .duration_ms]
| #csv
' myfile.json
I have the following JSON data:
{
"status": "ok",
"ok": true,
"data": "MFR-L",
"stores": [{
"name": "KOLL",
"lat": 52.93128,
"lng": 6.962956,
"dist": 1,
"x10": 1.129,
"isOpen": true
},
{
"name": "Takst",
"lat": 52.9523773,
"lng": 6.981644,
"dist": 1.3,
"x10": 1.809,
"isOpen": false
}]
}
I'm trying to convert it to a flat file using JQ, but I keep running into all sorts of problems, especially because of the file types ("cannot index boolean with string", etc).
This post has helped me flatten the contents of the array so far, like this:
jq -r -s 'map(.stores | map({nm: .name, lt: .lat} | [.nm, .lt])) | add [] | #csv
How can I get the contents higher up in the hierarchy to map to the array contents?
You could always collect the values you want from the parent objects separately from the child objects and combine them later.
e.g.,
$ jq -r '[.data] + (.stores[] | [.name, .lat, .lng, .dist]) | #csv' input.json
yields
"MFR-L","KOLL",52.93128,6.962956,1
"MFR-L","Takst",52.9523773,6.981644,1.3
There are several ways in which the illustrative JSON might be "flattened" (e.g. to CSV), but the following two approaches may be of interest. (I've omitted the invocation of #csv for ease-of-reading.)
$ jq '[.data, .stores[][]]' in.json
[
"MFR-L",
"KOLL",
52.93128,
6.962956,
1,
1.129,
true,
"Takst",
52.9523773,
6.981644,
1.3,
1.809,
false
]
$ jq '.data as $data | .stores[] | [$data, .[]]' in.json
[
"MFR-L",
"KOLL",
52.93128,
6.962956,
1,
1.129,
true
]
[
"MFR-L",
"Takst",
52.9523773,
6.981644,
1.3,
1.809,
false
]
Here is another approach which uses jq variables and string interpolation:
.data as $data
| .stores[]
| "\($data),\(.name),\(.lat),\(.lng),\(.dist),\(.x10),\(.isOpen)"
output with sample data:
"MFR-L,KOLL,52.93128,6.962956,1,1.129,true"
"MFR-L,Takst,52.9523773,6.981644,1.3,1.809,false"