I'm trying to come up with the correct jq syntax to convert json to csv.
Desired results:
<email>,<id>,<name>
e.g.
user1#whatever.nevermind.no,0,general
user2#whatever.nevermind.no,0,general
user1#whatever.nevermind.no,1,local
...
note that also need to ignore objects with empty "agent_priorities"
Input
[
{
"id": 0,
"name": "General",
"agent_priorities": {
"user1#whatever.nevermind.no": "normal",
"user2#whatever.nevermind.no": "normal"
}
},
{
"id": 1,
"name": "local",
"agent_priorities": {
"user1#whatever.nevermind.no": "normal"
}
},
{
"id": 2,
"name": "Engineering",
}
]
The following variant of the accepted answer checks for the existence of the "agent_priorities" key as per the requirements, and uses keys_unsorted to preserve the order of the keys:
jq -r '
.[]
| select(has("agent_priorities"))
| .id as $id
| .name as $name
| .agent_priorities
| keys_unsorted[]
| [., $id, $name ]
| #csv
' file.json
Store the id and name in variables, then iterate over the keys of agent_priorities:
jq -r '.[]
| .id as $id
| .name as $name
| .agent_priorities
| keys
| .[]
| [., $id, $name ]
| #csv
' file.json
Related
Given the following JSON
{
"tags": [
{
"key": "env",
"value": "foo"
},
{
"key": "env",
"value": "bar"
}
]
}
I am trying to find out the first tag where the key is env. I have this-
.tags[] | select (.key=="env") |.[0]
but that gives me an error Cannot index object with number
Use first(expr) to provide an expression that satisfies your usecase.
first(.tags[]? | select(.key == "env") .value)
You could wrap the results of your query in an array and then pick the first one
[.tags[] | select(.key=="env")] | .[0]
jq -r 'first( .tags[] | select(.key=="env") ).value'
jqplay
.tags[] flattens the array into a stream of values. You're applying .[0] to each of the values, not a filtered array. To filter an array, you'd use
.tags | map(select(...)) | .[0]
or
.tags | map(select(...)) | first
map(...) is a shorthand for [ .[] | ... ], so the above is equivalent to
.tags | [ .[] | select(...) ] | first
and
[ .tags[] | select(...) ] | first
Finally, [ ... ] | first can be written as first(...).
first( .tags[] | select(...) )
I'm trying to get a .csv out that includes occasional empty values.
Calling this API (https://www.campaignmonitor.com/api/subscribers/#getting-subscribers-details) I get the following:
[
{
"ID": "fc0ce7105baeaf97f47c99be31d02a91",
"Type": "Campaign",
"Name": "Campaign One",
"Actions": [
{
"Event": "Open",
"Date": "2010-10-12 13:18:00",
"IPAddress": "192.168.126.87",
"Detail": ""
},
{
"Event": "Click",
"Date": "2010-10-12 13:16:00",
"IPAddress": "192.168.126.87",
"Detail": "https://example.com/post/12323/"
}
]
}
{
"ID": "dsadsamdkl9309ujd432",
"Type": "Campaign",
"Name": "Campaign Two",
"Actions": []
}
]
What I want to get as output:
"Campaign One","Open"
"Campaign One","Click"
"Campaign Two","none"
What I currently get
"Campaign One","Open"
"Campaign One","Click"
I can't seem to find a way to include values when "Actions" == []
What I tried so far:
Attempt 1:
curl -u "apikey:x" https://api.createsend.com/api/v3.2/subscribers/listID/history.json?email=example#email.com | jq -r '.[] | .Name as $n | .Actions[] | ([$n, .Event | if . == null then "none" else . end]) | #csv'
Attempt 2:
curl -u "apikey:x" https://api.createsend.com/api/v3.2/subscribers/listID/history.json?email=example#email.com | jq -r '.[] | .Name as $n | .Actions[] | ([$n, .Event // "none"]) | #csv'
Attempt 3:
curl -u "apikey:x" https://api.createsend.com/api/v3.2/subscribers/listID/history.json?email=example#email.com | jq -r '.[] | .Name as $n | .Actions[] |.Actions[] | if . == [] then .Actions[].Event = "" else . end | ([$n, .Event]) | #csv'
With the alternative operator //:
jq -r '.[] | (.Actions[].Event // "none") as $e | [ .Name, $e ] | #csv'
This assumes that the missing comma on line 20 hast been inserted.
Input json:
{
"food_group": "fruit",
"glycemic_index": "low",
"fruits": {
"fruit_name": "apple",
"size": "large",
"color": "red"
}
}
Below two jq commands work:
# jq -r 'keys_unsorted[] as $key | "\($key), \(.[$key])"' food.json
food_group, fruit
glycemic_index, low
fruits, {"fruit_name":"apple","size":"large","color":"red"}
# jq -r 'keys_unsorted[0:2] as $key | "\($key)"' food.json
["food_group","glycemic_index"]
How to get values for the first two keys using jq in the same manner? I tried below
# jq -r 'keys_unsorted[0:2] as $key | "\($key), \(.[$key])"' food.json
jq: error (at food.json:9): Cannot index object with array
Expected output:
food_group, fruit
glycemic_index, low
To iterate over a hash array , you can use to_entries and that will transform to a array .
After you can use select to filter rows you want to keep .
jq -r 'to_entries[]| select( ( .value | type ) == "string" ) | "\(.key), \(.value)" '
You can use to_entries
to_entries[] | select(.key=="food_group" or .key=="glycemic_index") | "\(.key), \(.value)"
Demo
https://jqplay.org/s/Aqvos4w7bo
I previously got some help on here for some jq to csv issues. I ran into an issue where a few json files had some extra values that breaks the jq command
Here is the json data. The repairs section is what breaks the jq command
[
{
"Name": "John Doe",
"Car": [
"Car1",
"Car2"
],
"Location": "Texas",
"Repairs: {
"RepairLocations": {
"RepairsCompleted":[
"Fix1",
"Fix2"
]
}
}
},
{
"Name": "Jane Roe",
"Car": "Car1",
"Location": [
"Illinois",
"Kansas"
]
}
]
Here is the command
def expand($keys):
. as $in
| reduce $keys[] as $k ( [{}];
map(. + {
($k): ($in[$k] | if type == "array" then .[] else . end)
})
) | .[];
(.[0] | keys_unsorted) as $h
| $h, (.[] | expand($h) | [.[$h[]]]) | #csv
This is the end result i am trying to get. This data isnt actual data.
Name,Car,Location,Repairs:RepairLocation
John Doe,Car1,Texas,RepairsCompleted:Fix1
John Doe,Car1,Texas,RepairsCompleted:Fix2
John Doe,Car2,Texas,RepairsCompleted:Fix1
John Doe,Car2,Texas,RepairsCompleted:Fix2
Jane Roe,Car1,Illinois,
Jane Roe,Car1,Kansas,
Any advice on this would be great. I am struggling to figure jq out
A simple solution can be obtained using the same technique shown in one of the answers to the similar question you already asked. The only difference is fulfilling your requirements in the case where the "Repairs" key does not exist:
["Name", "Car", "Location", "Repairs:RepairLocation"],
(.[]
| [.Name]
+ (.Car|..|scalars|[.])
+ (.Location|..|scalars|[.])
+ (.Repairs|..|scalars
| [if . == null then . else "RepairsCompleted:\(.)" end]) )
| #csv
Avoiding the repetition with a helper function
def s: .. | scalars | [.];
["Name", "Car", "Location", "Repairs:RepairLocation"],
(.[]
| [.Name]
+ (.Car|s)
+ (.Location|s)
+ (.Repairs|s|map(if . == null then . else "RepairsCompleted:\(.)" end)))
| #csv
I've following json,
{
"A": {
"C": {
"D": "T1",
"E": 1
},
"F": {
"D": "T2",
"E": 2
}
},
"B": {
"C": {
"D": "T3",
"E": 3
}
}
}
I want to convert it into csv as follows,
A,C,T1,1
A,F,T2,2
B,C,T3,3
Description of output: The parents keys will be printed until, I've reached the leaf child. Once I reached leaf child, print its value.
I've tried following and couldn't succeed,
cat my.json | jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $rows[] | #csv'
and it throwing me an error.
I can't hardcode the parent keys, as the actual json has too many records. But the structure of the json is similar. What am I missing?
Some of the requirements are unclear, but the following solves one interpretation of the problem:
paths as $path
| {path: $path, value: getpath($path)}
| select(.value|type == "object" )
| select( [.value[]][0] | type != "object")
| .path + ([.value[]])
| #csv
(This program could be optimized but the presentation here is intended to make the separate steps clear.)
Invocation:
jq -r -f leaves-to-csv.jq input.json
Output:
"A","C","T1",1
"A","F","T2",2
"B","C","T3",3
Unquoted strings
To avoid the quotation marks around strings, you could replace the last component of the pipeline above with:
join(",")
Here is a solution using tostream and group_by
[
tostream
| select(length == 2) # e.g. [["A","C","D"],"T1"]
| .[0][:-1] + [.[1]] # ["A","C","T1"]
]
| group_by(.[:-1]) # [[["A","C","T1"],["A","C",1]],...
| .[] # [["A","C","T1"],["A","C",1]]
| .[0][0:2] + map(.[-1]|tostring) # ["A","C","T1","1"]
| join(",") # "A,C,T1,1"