use jq to format json data into csv data - json

{
"Users": [
{
"Attributes": [
{
"Name": "sub",
"Value": "1"
},
{
"Name": "phone_number",
"Value": "1234"
},
{
"Name": "referral_code",
"Value": "abc"
}
]
},
{
"Attributes": [
{
"Name": "sub",
"Value": "2"
},
{
"Name": "phone_number",
"Value": "5678"
},
{
"Name": "referral_code",
"Value": "def"
}
]
}
]
}
How can I produce output like below ?
1,1234,abc
2,5678,def
jq '.Users[] .Attributes[] .Value' test.json
produces
1
1234
abc
2
5678
def

Not sure this is the cleanest way to handle this, but the following will get the desired output:
.Users[].Attributes | map(.Value) | #csv
Loop through all the deep Attributes .Users[].Attributes
map() to get all the Value's
Convert to #csv
jqPlay demo

If you don't need the output to be guaranteed to be CSV, and if you're sure the "Name" values are presented in the same order, you could go with:
.Users[].Attributes
| from_entries
| [.[]]
| join(",")
To be safe though it would be better to ensure consistency of ordering:
(.Users[0] | [.Attributes[] | .Name]) as $keys
| .Users[]
| .Attributes
| from_entries
| [.[ $keys[] ]]
| join(",")
Using join(",") will produce the comma-separated values as shown in the Q (without the quotation marks), but is not guaranteed to produce the expected CSV for all valid values of the input. If you don't mind the pesky quotation marks, you could use #csv, or if you want to skip the quotation marks around all numeric values:
map(tonumber? // .) | #csv

Related

How to parse nested json to csv using command line

I want to parse a nested json to csv. The data looks similar to this.
{"tables":[{"name":"PrimaryResult","columns":[{"name":"name","type":"string"},{"name":"id","type":"string"},{"name":"custom","type":"dynamic"}]"rows":[["Alpha","1","{\"age\":\"23\",\"number\":\"xyz\"}]]]}
I want csv file as:
name id age number
alpha 1 23 xyz
I tried:
jq -r ".tables | .[] | .columns | map(.name)|#csv" demo.json > demo.csv
jq -r ".tables | .[] | .rows |.[]|#csv" demo.json >> demo.csv
But I am not getting expected result.
Output:
name id custom
alpha 1 {"age":"23","number":"xyz}
Expected:
name id age number
alpha 1 23 xyz
Assuming valid JSON input:
{
"tables": [
{
"name": "PrimaryResult",
"columns": [
{ "name": "name", "type": "string" },
{ "name": "id", "type": "string" },
{ "name": "custom", "type": "dynamic" }
],
"rows": [
"Alpha",
"1",
"{\"age\":\"23\",\"number\":\"xyz\"}"
]
}
]
}
And assuming fixed headers:
jq -r '["name", "id", "age", "number"],
(.tables[].rows | [.[0,1], (.[2] | fromjson | .age, .number)])
| #csv' input.json
Output:
"name","id","age","number"
"Alpha","1","23","xyz"
If any of the assumptions is wrong, you need to clarify your requirements, e.g.
How are column names determined?
What happens if the input contains multiple tables?
As the "dynamic" object always of the same shape? Or can it sometimes contain fewer, more, or different columns?
Assuming that the .rows array is a 2D array of rows and fields, and that a column of type "dynamic" always expects a JSON-encoded object whose fields represent further columns but may or may not always be present in every row.
Then you could go with transposing the headers array and the rows array in order to integratively process each column by their type, especially collecting all keys from the "dynamic" type on the fly, and then transpose it back to get the row-based CSV output.
Input (I have added another row for illustration):
{
"tables": [
{
"name": "PrimaryResult",
"columns": [
{
"name": "name",
"type": "string"
},
{
"name": "id",
"type": "string"
},
{
"name": "custom",
"type": "dynamic"
}
],
"rows": [
[
"Alpha",
"1",
"{\"age\":\"23\",\"number\":\"123\"}"
],
[
"Beta",
"2",
"{\"age\":\"45\",\"word\":\"xyz\"}"
]
]
}
]
}
Filter:
jq -r '
.tables[] | [.columns, .rows[]] | transpose | map(
if first.type == "string" then first |= .name
elif first.type == "dynamic" then
.[1:] | map(fromjson)
| (map(keys[]) | unique) as $keys
| [$keys, (.[] | [.[$keys[]]])] | transpose[]
else empty end
)
| transpose[] | #csv
'
Output:
"name","id","age","number","word"
"Alpha","1","23","123",
"Beta","2","45",,"xyz"
Demo

Extract certain array elements with its parent keys using jq

I would like to know how to use jq to extract patterns from a .json file
echo '{"parts": [{"name":"core","items":"garbage with ITEM1 ITEM2 and more"},{"name":"misc","items":"ITEM3 ITEM4 ITEM5 bla bla"} ]}' | jq '.parts | .[] | .items |=split(" ")'
{
"name": "core",
"items": [
"garbage",
"with",
"ITEM1",
"ITEM2",
"and",
"more"
]
}
{
"name": "misc",
"items": [
"ITEM3",
"ITEM4",
"ITEM5",
"bla",
"bla"
]
}
I think in splitting the items, but I don't know how to extract each ITEMx.
I want to obtain this output:
{ "core","ITEM1" }
{ "core","ITEM2" }
{ "misc","ITEM3" }
{ "misc","ITEM4" }
{ "misc","ITEM5" }
Your desired output is not valid JSON.
Do you want the words form an array under the value of the .name field?
jq '.parts[] | {(.name): (.items | split(" "))}'
{
"core": [
"garbage",
"with",
"ITEM1",
"ITEM2",
"and",
"more"
]
}
{
"misc": [
"ITEM3",
"ITEM4",
"ITEM5",
"bla",
"bla"
]
}
Demo
Or do you want each word to form a separate object?
jq '.parts[] | (.items | split(" "))[] as $word | {(.name): $word}'
{"core":"garbage"}
{"core":"with"}
{"core":"ITEM1"}
{"core":"ITEM2"}
{"core":"and"}
{"core":"more"}
{"misc":"ITEM3"}
{"misc":"ITEM4"}
{"misc":"ITEM5"}
{"misc":"bla"}
{"misc":"bla"}
Demo
To only capture words that match the regex ITEM\d+, you could employ the scan function instead of splitting:
jq '.parts[] | {(.name): .items | scan("ITEM\\d+")}'
{"core":"ITEM1"}
{"core":"ITEM2"}
{"misc":"ITEM3"}
{"misc":"ITEM4"}
{"misc":"ITEM5"}
Demo
Building on your attempt, we could try:
.parts
| .[]
| .items |= (split(" ") | map(select(test("ITEM"))))
| {(.name): .items[]}
This produces a stream of objects such as {"core":"ITEM1"}. If you really want the non-JSON output shown in the Q, it's easy enough to add the additional step.

Combine JSON Field and add values using jq

I have to aggregate a few JSON results from a site. Because the site has a query concurrency limit and the queries timeout, the time frame for the queries have to be divided. So I am left with a JSON as follows:
{
"results": [
[
{
"field": "AccountId",
"value": "11352"
},
{
"field": "number_of_requests",
"value": "241398"
}
],
[
{
"field": "AccountId",
"value": "74923"
},
{
"field": "number_of_requests",
"value": "238566"
}
]
],
"statistics": {
"recordsMatched": 502870.0,
"recordsScanned": 165908292.0,
"bytesScanned": 744173091162.0
},
"status": "Complete"
}
{
"results": [
[
{
"field": "AccountId",
"value": "11352"
},
{
"field": "number_of_requests",
"value": "185096"
}
]
],
"statistics": {
"recordsMatched": 502870.0,
"recordsScanned": 165908292.0,
"bytesScanned": 744173091162.0
},
"status": "Complete"
}
I need to aggregate the results, match the values to the number of requests and print out the result in descending Order.
Desired Output:
AccountID : Number of Requests
11352 : 426494
74923 : 238566
Current Output:
AccountID : Number of Requests
11352 : 241398
11352 : 185096
74923 : 238566
The jq query I am running currently takes the file name as ResultDir:
list=$(jq -S '.results[] | map( { (.field) : .value} ) | add ' $ResultsDir |
jq -s -c 'sort_by(.number_of_requests|tonumber) | reverse[] ' |
jq -r '"\(.AccountId) : \(.number_of_requests)"')
How do I combine the results of the same accounts before printing it out? The results also need to be in descending order of number of requests.
When possible, it's generally advisable to minimize the number of calls to jq. In this case, it's easy enough to achieve the desired output with just one call to jq.
Assuming the input is a valid stream of JSON objects along the lines shown in the Q, the following produces the desired output:
jq -nr '
[inputs | .results[] | map( { (.field) : .value} ) | add]
| group_by(.AccountId)
| map([.[0].AccountId, (map(.number_of_requests|tonumber) | add)])
| sort_by(.[1]) | reverse
| .[]
| join(" : ")
'

Extract inner array matching values using jq

My json is as shown below:
[
[
{
"id": "abcd"
},
{
"address": [
"140 Deco st"
]
}
],
[
{
"id": "xyz"
},
{
"dummy": "This is dummy"
}
],
[
{
"id": "12356"
},
{
"address": [
"140 Deco st"
]
}
]]
Now, I want to capture only those ids who have dummy value of "This is dummy". Some of the data may or may not have dummy and address fields.
I tried below but it gave me error "... cannot have their containment checked"
jq -c '.[] | .[] | select(.dummy | contains("This is dummy")) | .[] | .id'
Any help is much appreciated!
contains is quite tricky to use correctly. Since the requirement is:
to capture only those ids who have dummy value of "This is dummy"
I would suggest:
.[]
| select( any(.[]; .dummy == "This is dummy") )
| add
| .id
or perhaps (depending on your detailed requirements):
.[]
| select( any(.[]; .dummy == "This is dummy") )
| .[]
| .id? // empty

jq: Insert values according to mappings from external file

I was wondering how I can complete this task by command line jq. I make up a file with similar nested structure as follows:
{
"item": "item1",
"features": [
{
"feature": "feature_a",
"value": ""
},
{
"feature": "feature_b",
"value": ""
}
]
}
Now I have another file that maps the feature to value:
feature_a value_1
feature_b value_2
So I would like to insert the value into the first json file, according to the maps, resulting the following output:
{
"item": "item1";
"features": [
{
"feature": "feature_a",
"value": "value_1"
},
{
"feature": "feature_b",
"value": "value_2"
}
]
}
How I can achieve above operation by jq?
Thanks in advance!
Assuming the text file is in dict.txt and the JSON file is in source.json, the invocation
jq -Rs --argfile target source.json dict.txt '
([ split("\n")[]
| select(length>0)
| split(" ")
| { (.[0]): .[1]} ]
| add) as $dict
| $target
| .features |= map(.value = $dict[.feature])'
would yield the desired output.
The main reason for including select(length>0) is to skip any empty strings that might result from using split("\n") to split an entire file.