I'm trying to convert a json object into csv via jq. This is the json structure:
{
"totalCount": 4440,
"data": [
{
"company": {
"name": "My_company_name",
"countryCode": "US",
"portfolioName": "My_portfolio"
},
"eventDate": "2021-11-22T00:00:00",
"newValue": null,
"oldValue": null,
"ruleCode": 704,
"ruleName": "New Accounts",
"summary": "Explanations..."
},
{
"company": {
"name": "My_company_name 2",
"countryCode": "UK",
"portfolioName": "My_portfolio"
},
"eventDate": "2021-10-22T00:00:00",
"newValue": null,
"oldValue": null,
"ruleCode": 701,
"ruleName": "Hello",
"summary": "otherExplanations..."
}
...
]
}
For data in "first level", I've no problems:
jq -r '.data | map({eventDate, ruleCode, ruleName, summary, oldValue, newValue}) | (first | keys_unsorted) as $keys | map([to_entries[] | .value]) as $rows | $keys,$rows[] | #csv' input.json > output.csv
But I'ld like to add the company name and country code for example, and I don't kown to do this, with this king of data in second "level".
I'ld to obtain something like that:
"eventDate","ruleCode","ruleName","summary","oldValue","newValue","companyName", "companyCountryCode"
"2021-11-22T00:00:00",704,"New Accounts","Explanations...",,,"My_company_name", "US"
"2021-11-22T00:00:00",701,"Hello","otherExplanations...",,,"My_company_name 2", "UK"
Could you help me ?
Thanks
If you don't mind, I added a . to divide top-level from sub-level headers to make things easier (namely company.name and company.countryCode):
jq --raw-output '[
"eventDate",
"ruleCode",
"ruleName",
"summary",
"oldValue",
"newValue",
"company.name",
"company.countryCode"
] as $h
| $h, (.data[] | [getpath($h[] / ".")])
| #csv'
"eventDate","ruleCode","ruleName","summary","oldValue","newValue","company.name","company.countryCode"
"2021-11-22T00:00:00",704,"New Accounts","Explanations...",,,"My_company_name","US"
"2021-10-22T00:00:00",701,"Hello","otherExplanations...",,,"My_company_name 2","UK"
Demo
Related
I am trying to out all the data from my json file that matches the value "data10=true" it does that but only grabs the names, how can i make it so it will output everything in my json file with anything that matches the "data10=true"?
this is what ive got data=$(jq -c 'to_entries[] | select (.value.data10 == "true")| [.key, .value.name]' data.json )
This is in my YAML template btw, running it as a pipeline in devops.
The detailed requirements are unclear, but hopefully you'll be able to use the following jq program as a guide:
..
| objects
| select( .data10 == "true" )
| to_entries[]
| select(.key != "data10")
| [.key, .value]
This will recursively (thanks to the initial ..) examine all the JSON objects in the input.
p.s.
If you want to make the selection based on whether .data10 is "true" or true, you could change the criterion to .data10 | . == true or . == "true".
jq 'to_entries | map(select(.value.data10=="true")) | from_entries' data.json
input data.json,
with false value:
{
"FOO": {
"data10": "false",
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"data10": "true",
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"data10": "true",
"name": "Jack",
"location": "Whereever"
}
}
output:
{
"BAR": {
"data10": "true",
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"data10": "true",
"name": "Jack",
"location": "Whereever"
}
}
based on: https://stackoverflow.com/a/37843822/983325
I need to print some results with jq to take json.
This is an example:
{
"data": [
{
"time": 20201606,
"event": {
"ip": "127.0.1",
"hostname": "srv1",
"locations": [
"UK",
"site1"
],
"num": 1
}
},
{
"time": 202016034,
"event": {
"ip": "127.0.2",
"hostname": "srv2",
"locations": [
"UK",
"site2"
],
"num": 3
}
}
]
}
Like to generate this output "num, ip, hostname, locations":
1, srv1, 127.0.1, UK,site1
2, srv2, 127.0.2, HK,site2
3, srv3, 127.0.3, LO,site3
How can I print this via jq?
Join locations by a comma, and put the result into an array with other fields. Then join again by a comma followed by a space to get the desired output format. E.g.:
.data[].event | [
.num,
.hostname,
.ip,
(.locations | join(",")) ?
] | join(", ")
Use --raw-output/-r option in the command line invocation to get raw strings instead of JSON strings.
Online demo
At its core, you want to build an array consisting of the values you want:
$ jq '.data[].event | [.num, .hostame, .ip, .locations]' tmp.json
[
1,
null,
"127.0.1",
[
"UK",
"site1"
]
]
[
3,
null,
"127.0.2",
[
"UK",
"site2"
]
]
From there, it's a matter of formatting. First, let's turn the list of locations into a single string:
$ jq '.data[].event | [.num, .hostame, .ip, (.locations|join(","))]' tmp.json
[
1,
null,
"127.0.1",
"UK,site1"
]
[
3,
null,
"127.0.2",
"UK,site2"
]
Next, let's join those strings into a ", "-separated string.
$ jq '.data[].event | [.num, .hostame, .ip, (.locations|join(","))] | join(", ")' tmp.json
"1, , 127.0.1, UK,site1"
"3, , 127.0.2, UK,site2"
Finally, you can use the -r flag to output raw text rather than a JSON string value.
$ jq -r '.data[].event | [.num, .hostame, .ip, (.locations|join(","))] | join(", ")' tmp.json
1, , 127.0.1, UK,site1
3, , 127.0.2, UK,site2
i have two json files structured like that:
file 1
[
{
"id": 25422,
"location": "Hotel X",
"suppliers": [
12
]
},
{
"id": 25423,
"location": "Hotel Y",
"suppliers": [
13
]
}]
file 2
[
{
"id": 12,
"vatNumber": "0000000000"
},
{
"id": 14,
"vatNumber": "0000000001"
}]
and i'd like a result like this
[
{
"id": 25422,
"location": "Hotel X",
"suppliers": [
12
],
"vatNumber": "0000000000"
},
{
"id": 25423,
"location": "Hotel Y",
"suppliers": [
13
],
}]
The important thing to me is that the matching vatNumbers, are set in the first file. Supplier arrays are not required anymore after the melding, if it simplifies the job.
Also jq is not essential, but i need something i can use via terminal to set up a script.
Thank you in advance.
Here's one of many possible solutions. If your jq does not have INDEX/2, then either upgrade your jq or include its def (available e.g. from https://github.com/stedolan/jq/blob/master/src/builtin.jq):
Invocation:
jq -n --argfile f1 file1.json --argfile f2 file2.json -f merge.jq
merge.jq:
INDEX($f2[] ; .id) as $dict
| $f1
| map( ($dict[.suppliers[0]|tostring]|.vatNumber) as $vn
| if $vn then .vatNumber = $vn else . end)
I've started using jq just very recently and I would like to know if something like this is even possible.
Example:
{
"name": "device",
"version": "1.0.0",
"address": [
{
"address": "10.1.2.3",
"interface": "wlan1_wifi"
},
{
"address": "10.1.2.5",
"interface": "wlan2_link"
},
{
"address": "10.1.2.4",
"interface": "ether1"
}
],
"wireless": [
{
"name": "wlan1_wifi",
"type": "5Ghz",
"ssid": "wifi"
},
{
"name": "wlan2_link",
"type": "2Ghz",
"ssid": "link"
}
]
}
Firstly let's transform the example to this json object:
cat json | jq '. | {"name": ."name", "version": ."version", "wireless": [."wireless"[] | {"name": ."name", "type": ."type", "ssid": ."ssid"}]}'
{
"name": "device",
"version": "1.0.0",
"wireless": [
{
"name": "wlan1_wifi",
"type": "5Ghz",
"ssid": "wifi"
},
{
"name": "wlan2_link",
"type": "2Ghz",
"ssid": "link"
}
]
}
Now there's a problem. I need to assign an address to the "wireless" array. The address is stored in "address" array.
So the question: is there a way of finding the right json object in "address" based on "name" (in wireless array) and "interface" (in address array) for every json object in "wireless" array and then assigning "address" to it?
The final result should look like this:
{
"name": "device",
"version": "1.0.0",
"wireless": [
{
"name": "wlan1_wifi",
"type": "5Ghz",
"ssid": "wifi",
"address": "10.1.2.3"
},
{
"name": "wlan2_link",
"type": "2Ghz",
"ssid": "link",
"address": "10.1.2.5"
}
]
}
Answer:
Here's my answer based on the answer from #peak. Instead of copying the content of .wireless and then using map, I'm cherry picking the keys that I want to include only. This also allows me to position "address" how ever I want.
(INDEX(.address[]; .interface)) as $dict
| {name: .name, version: .version,
wireless: [.wireless[] | {name, address: ($dict[.name]|.address), type, ssid}]}
The following produces the output as originally requested:
(.wireless[].name) as $name
| .address[]
| select(.interface == $name)
| { wireless: {name: $name, address}}
However the above filter could potentially produce more than one result, so you might want to make modifications accordingly.
Revised revised requirements
If your jq has INDEX/2 (which was only made available AFTER jq 1.5 was released), you can simply use it to create a lookup table:
(INDEX(.address[]; .interface)) as $dict
| {name,
version,
wireless: (.wireless
| map(. + {address: ($dict[.name]|.address) }) ) }
Or (depending perhaps on the exact requirements):
(INDEX(.address[]; .interface)) as $dict
| del(.address)
| .wireless |= map(. + {address: ($dict[.name]|.address) })
If your jq does not have INDEX/2, then you could easily adapt the above (using reduce), or even more easily snarf the def of INDEX/2 from https://github.com/stedolan/jq/blob/master/src/builtin.jq
Giving a JSON string like this,
[
{
"id": 1,
"name": "Arthur",
"age": "21"
},
{
"id": 2,
"name": "Richard",
"age": "32"
}
]
How to filter by name and get the age?
E.g., given the name being "Richard", let jq return "32". Thx.
$ jq --arg name Richard '.[] | select(.name==$name) | .age' input.json
"32"
When using jq like this in Windows, the quoting would have to be appropriate for Windows.