Related
I want to parse a nested json to csv. The data looks similar to this.
{"tables":[{"name":"PrimaryResult","columns":[{"name":"name","type":"string"},{"name":"id","type":"string"},{"name":"custom","type":"dynamic"}]"rows":[["Alpha","1","{\"age\":\"23\",\"number\":\"xyz\"}]]]}
I want csv file as:
name id age number
alpha 1 23 xyz
I tried:
jq -r ".tables | .[] | .columns | map(.name)|#csv" demo.json > demo.csv
jq -r ".tables | .[] | .rows |.[]|#csv" demo.json >> demo.csv
But I am not getting expected result.
Output:
name id custom
alpha 1 {"age":"23","number":"xyz}
Expected:
name id age number
alpha 1 23 xyz
Assuming valid JSON input:
{
"tables": [
{
"name": "PrimaryResult",
"columns": [
{ "name": "name", "type": "string" },
{ "name": "id", "type": "string" },
{ "name": "custom", "type": "dynamic" }
],
"rows": [
"Alpha",
"1",
"{\"age\":\"23\",\"number\":\"xyz\"}"
]
}
]
}
And assuming fixed headers:
jq -r '["name", "id", "age", "number"],
(.tables[].rows | [.[0,1], (.[2] | fromjson | .age, .number)])
| #csv' input.json
Output:
"name","id","age","number"
"Alpha","1","23","xyz"
If any of the assumptions is wrong, you need to clarify your requirements, e.g.
How are column names determined?
What happens if the input contains multiple tables?
As the "dynamic" object always of the same shape? Or can it sometimes contain fewer, more, or different columns?
Assuming that the .rows array is a 2D array of rows and fields, and that a column of type "dynamic" always expects a JSON-encoded object whose fields represent further columns but may or may not always be present in every row.
Then you could go with transposing the headers array and the rows array in order to integratively process each column by their type, especially collecting all keys from the "dynamic" type on the fly, and then transpose it back to get the row-based CSV output.
Input (I have added another row for illustration):
{
"tables": [
{
"name": "PrimaryResult",
"columns": [
{
"name": "name",
"type": "string"
},
{
"name": "id",
"type": "string"
},
{
"name": "custom",
"type": "dynamic"
}
],
"rows": [
[
"Alpha",
"1",
"{\"age\":\"23\",\"number\":\"123\"}"
],
[
"Beta",
"2",
"{\"age\":\"45\",\"word\":\"xyz\"}"
]
]
}
]
}
Filter:
jq -r '
.tables[] | [.columns, .rows[]] | transpose | map(
if first.type == "string" then first |= .name
elif first.type == "dynamic" then
.[1:] | map(fromjson)
| (map(keys[]) | unique) as $keys
| [$keys, (.[] | [.[$keys[]]])] | transpose[]
else empty end
)
| transpose[] | #csv
'
Output:
"name","id","age","number","word"
"Alpha","1","23","123",
"Beta","2","45",,"xyz"
Demo
I would like to know how to use jq to extract patterns from a .json file
echo '{"parts": [{"name":"core","items":"garbage with ITEM1 ITEM2 and more"},{"name":"misc","items":"ITEM3 ITEM4 ITEM5 bla bla"} ]}' | jq '.parts | .[] | .items |=split(" ")'
{
"name": "core",
"items": [
"garbage",
"with",
"ITEM1",
"ITEM2",
"and",
"more"
]
}
{
"name": "misc",
"items": [
"ITEM3",
"ITEM4",
"ITEM5",
"bla",
"bla"
]
}
I think in splitting the items, but I don't know how to extract each ITEMx.
I want to obtain this output:
{ "core","ITEM1" }
{ "core","ITEM2" }
{ "misc","ITEM3" }
{ "misc","ITEM4" }
{ "misc","ITEM5" }
Your desired output is not valid JSON.
Do you want the words form an array under the value of the .name field?
jq '.parts[] | {(.name): (.items | split(" "))}'
{
"core": [
"garbage",
"with",
"ITEM1",
"ITEM2",
"and",
"more"
]
}
{
"misc": [
"ITEM3",
"ITEM4",
"ITEM5",
"bla",
"bla"
]
}
Demo
Or do you want each word to form a separate object?
jq '.parts[] | (.items | split(" "))[] as $word | {(.name): $word}'
{"core":"garbage"}
{"core":"with"}
{"core":"ITEM1"}
{"core":"ITEM2"}
{"core":"and"}
{"core":"more"}
{"misc":"ITEM3"}
{"misc":"ITEM4"}
{"misc":"ITEM5"}
{"misc":"bla"}
{"misc":"bla"}
Demo
To only capture words that match the regex ITEM\d+, you could employ the scan function instead of splitting:
jq '.parts[] | {(.name): .items | scan("ITEM\\d+")}'
{"core":"ITEM1"}
{"core":"ITEM2"}
{"misc":"ITEM3"}
{"misc":"ITEM4"}
{"misc":"ITEM5"}
Demo
Building on your attempt, we could try:
.parts
| .[]
| .items |= (split(" ") | map(select(test("ITEM"))))
| {(.name): .items[]}
This produces a stream of objects such as {"core":"ITEM1"}. If you really want the non-JSON output shown in the Q, it's easy enough to add the additional step.
{
"Users": [
{
"Attributes": [
{
"Name": "sub",
"Value": "1"
},
{
"Name": "phone_number",
"Value": "1234"
},
{
"Name": "referral_code",
"Value": "abc"
}
]
},
{
"Attributes": [
{
"Name": "sub",
"Value": "2"
},
{
"Name": "phone_number",
"Value": "5678"
},
{
"Name": "referral_code",
"Value": "def"
}
]
}
]
}
How can I produce output like below ?
1,1234,abc
2,5678,def
jq '.Users[] .Attributes[] .Value' test.json
produces
1
1234
abc
2
5678
def
Not sure this is the cleanest way to handle this, but the following will get the desired output:
.Users[].Attributes | map(.Value) | #csv
Loop through all the deep Attributes .Users[].Attributes
map() to get all the Value's
Convert to #csv
jqPlay demo
If you don't need the output to be guaranteed to be CSV, and if you're sure the "Name" values are presented in the same order, you could go with:
.Users[].Attributes
| from_entries
| [.[]]
| join(",")
To be safe though it would be better to ensure consistency of ordering:
(.Users[0] | [.Attributes[] | .Name]) as $keys
| .Users[]
| .Attributes
| from_entries
| [.[ $keys[] ]]
| join(",")
Using join(",") will produce the comma-separated values as shown in the Q (without the quotation marks), but is not guaranteed to produce the expected CSV for all valid values of the input. If you don't mind the pesky quotation marks, you could use #csv, or if you want to skip the quotation marks around all numeric values:
map(tonumber? // .) | #csv
I have a json structure that looks like this:
{
"lorry1": {
"box1": [
{"item": "shoes", "state": "new"},
{"item": "snacks", "state": "new"}
],
"box2": [
{"item": "beer", "state": "cold"},
{"item": "potatoes"}
]
},
"lorry2": {
"box1": [
{"item": "shoes", "state": "new"},
{"item": "snacks", "state": "new"}
],
"box2": [
{"item": "beer", "state": "lukewarm"}
]
}
}
Now I want to know where I can find shoes:
I could come up with this jq query:
to_entries | select(.[].value | .[][].item=="shoes") | map({"lorry": "\(.key)" })
But that only gives me the lorries. Useful, but not quite there yet. I'd like to know the box they're in as well.
I came up with this, but it obviously is not correct:
to_entries | select(.[].value | .[][].item=="shoes") | keys as $box |map({"lorry": "\(.key)", "box": $box })
The answer I'd like to get is lorry1, box1 and lorry2, box1.
Even better yet: I'd like to find all items and provide the information, like this:
"shoes": [ {"lorry1", "box1"}, {"lorry2", "box1" } ],
"snacks": [ {"lorry1", "box1"}, {"lorry2", "box1"} ],
"beer": [ {"lorry1", "box2"}, {"lorry2", "box2"} ],
"potatoes": [ {"lorry1", "box2"} ]
but that may be asking a bit too much :)
This looks like a overkill to me but it does the job.
[path(.[][][].item) as $p | [$p, getpath($p)]] |
group_by( .[1] ) |
map({(.[0][1]): (. | map([.[0][0,1]]))})|
add
Save the above jq filter in file item_in.jq and run it as jq --from-file item_in.jq. Passing your input to this gives the following output:
{
"beer": [
[
"lorry1",
"box2"
],
[
"lorry2",
"box2"
]
],
"potatoes": [
[
"lorry1",
"box2"
]
],
"shoes": [
[
"lorry1",
"box1"
],
[
"lorry2",
"box1"
]
],
"snacks": [
[
"lorry1",
"box1"
],
[
"lorry2",
"box1"
]
]
}
The initial transformation was to dump leaves and their paths from the input JSON tree.
See
https://github.com/stedolan/jq/issues/78#issuecomment-17819519
The answer I'd like to get is lorry1, box1 and lorry2, box1
In this case, you can get it with:
path(.. | select(.item? == "shoes"))
Which returns:
["lorry1","box1",0]
["lorry2","box1",0]
These are the paths in your object that will lead to an object which .item property is set to "shoes"
Here's a generic solution that does not assume the "items" are in arrays, or even that the values associated with the "item" keys are always strings:
jq -c '. as $in
| [paths as $p | select($p[-1] == "item") | $p]
| group_by(. as $p | $in|getpath($p))
| .[]
| (.[0] as $p | $in | getpath($p)) as $v
| {($v|tostring): ( map(.[:-1] | if .[-1] | type == "number" then .[:-1] else . end)) }
'
Output
With your input:
{"beer":[["lorry1","box2"],["lorry2","box2"]]}
{"potatoes":[["lorry1","box2"]]}
{"shoes":[["lorry1","box1"],["lorry2","box1"]]}
{"snacks":[["lorry1","box1"],["lorry2","box1"]]}
If you want the output as a single JSON object, then collect the above into an array and use add.
Using jq, I'd like to cherry-pick key/value pairs from the following json:
{
"project": "Project X",
"description": "This is a description of Project X",
"nodes": [
{
"name": "server001",
"detail001": "foo",
"detail002": "bar",
"networks": [
{
"net_tier": "network_tier_001",
"ip_address": "10.1.1.10",
"gateway": "10.1.1.1",
"subnet_mask": "255.255.255.0",
"mac_address": "00:11:22:aa:bb:cc"
}
],
"hardware": {
"vcpu": 1,
"mem": 1024,
"disks": [
{
"disk001": 40,
"detail001": "foo"
},
{
"disk002": 20,
"detail001": "bar"
}
]
},
"os": "debian8",
"geo": {
"region": "001",
"country": "Sweden",
"datacentre": "Malmo"
},
"detail003": "baz"
}
],
"detail001": "foo"
}
For the sake of an example, I'd like to parse the following keys and their values: "Project", "name", "net_tier", "vcpu", "mem", "disk001", "disk002".
I'm able to parse individual elements without much issue, but due to the hierarchical nature of the full parse, I've not had much luck parsing down different branches (i.e. both networks and hardware > disks).
Any help appreciated.
Edit:
For clarity, the output I'm going for is a comma-separated CSV. In terms of parsing all combinations, covering the sample data in the example will do for now. I will hopefully be able to expand on any suggestions.
Here is a different filter which computes the unique set of network tier and disk names and then generates a result with columns appropriate to the data.
{
tiers: [ .nodes[].networks[].net_tier ] | unique
, disks: [ .nodes[].hardware.disks[] | keys[] | select(startswith("disk")) ] | unique
} as $n
| def column_names($n): [ "project", "name" ] + $n.tiers + ["vcpu", "mem"] + $n.disks ;
def tiers($n): [ $n.tiers[] as $t | .networks[] | if .net_tier==$t then $t else null end ] ;
def disks($n): [ $n.disks[] as $d | map(select(.[$d]!=null)|.[$d])[0] ] ;
def rows($n):
.project as $project
| .nodes[]
| .name as $name
| tiers($n) as $tier_values
| .hardware
| .vcpu as $vcpu
| .mem as $mem
| .disks
| disks($n) as $disk_values
| [$project, $name] + $tier_values + [$vcpu, $mem] + $disk_values
;
column_names($n), rows($n)
| #csv
The benfit of this approach becomes apparent if we add another node to the sample data:
{
"name": "server002",
"networks": [
{
"net_tier": "network_tier_002"
}
],
"hardware": {
"vcpu": 1,
"mem": 1024,
"disks": [
{
"disk002": 40,
"detail001": "foo"
}
]
}
}
Sample Run (assuming filter in filter.jq and amended data in data.json)
$ jq -Mr -f filter.jq data.json
"project","name","network_tier_001","network_tier_002","vcpu","mem","disk001","disk002"
"Project X","server001","network_tier_001","",1,1024,40,20
"Project X","server002",,"network_tier_002",1,1024,,40
Try it online!
Here's one way you could achieve the desired output.
program.jq:
["project","name","net_tier","vcpu","mem","disk001","disk002"],
[.project]
+ (.nodes[] | .networks[] as $n |
[
.name,
$n.net_tier,
(.hardware |
.vcpu,
.mem,
(.disks | add["disk001","disk002"])
)
]
)
| #csv
$ jq -r -f program.jq input.json
"project","name","net_tier","vcpu","mem","disk001","disk002"
"Project X","server001","network_tier_001",1,1024,40,20
Basically, you'll want to project the fields that you want into arrays so you may convert those arrays to csv rows. Your input makes it seem like there could potentially be multiple networks for a given node. So if you wanted to output all combinations, that would have to be flattened out.
Here's another approach, that is short enough to speak for itself:
def s(f): first(.. | f? // empty) // null;
[s(.project), s(.name), s(.net_tier), s(.vcpu), s(.mem), s(.disk001), s(.disk002)]
| #csv
Invocation:
$ jq -r -f value-pairs.jq input.json
Result:
"Project X","server001","network_tier_001",1,1024,40,20
With headers
Using the same s/1 as above:
. as $d
| ["project", "name", "net_tier", "vcpu", "mem", "disk001","disk002"]
| (., map( . as $v | $d | s(.[$v])))
| #csv
With multiple nodes
Again with s/1 as above:
.project as $p
| ["project", "name", "net_tier", "vcpu", "mem", "disk001","disk002"] as $h
| ($h,
(.nodes[] as $d
| $h
| map( . as $v | $d | s(.[$v]) )
| .[0] = $p)
) | #csv
Output with the illustrative multi-node data:
"project","name","net_tier","vcpu","mem","disk001","disk002"
"Project X","server001","network_tier_001",1,1024,40,20
"Project X","server002","network_tier_002",1,1024,,40