How to make paths to leafs of a JSON? - json

Say we have the following JSON:
[
{
"dir-1": [
"file-1.1",
"file-1.2"
]
},
"dir-1",
{
"dir-2": [
"file-2.1"
]
}
]
And we want to get the next output:
"dir-1/file-1.1"
"dir-1/file-1.2"
"dir-1"
"dir-2/file-2.1"
i.e. to get the paths to all leafs, joining items with /. Is there a way to do that on JQ?
I tried something like this:
cat source-file | jq 'path(..) | [ .[] | tostring ] | join("/")'
But it doesn't produce what I need even close.

You could take advantage of how streams work by merging the path with their values. Streams will only emit path, value pairs for leaf values. Just ignore the numbered indices.
$ jq --stream '
select(length == 2) | [(.[0][] | select(strings)), .[1]] | join("/")
' source-file
returns:
"dir-1/file-1.1"
"dir-1/file-1.2"
"dir-1"
"dir-2/file-2.1"

Here is a solution similar to Jeff Mercado's which uses tostream and flatten
tostream | select(length==2) | .[0] |= map(strings) | flatten | join("/")
Try it online at jqplay.org
Another way is to use a recursive function to walk the input such as
def slashpaths($p):
def concat($p;$k): if $p=="" then $k else "\($p)/\($k)" end;
if type=="array" then .[] | slashpaths($p)
elif type=="object" then
keys_unsorted[] as $k
| .[$k] | slashpaths(concat($p;$k))
else concat($p;.) end;
slashpaths("")
Try it online at tio.run!

Using --stream is good but the following is perhaps less esoteric:
paths(scalars) as $p
| getpath($p) as $v
| ($p | map(strings) + [$v])
| join("/")
(If using jq 1.4 or earlier, and if any of the leaves might be numeric or boolean or null, then [$v] above should be replaced by [$v|tostring].)
Whether the result should be regarded as "paths to leaves" is another matter...

Related

Extract the key names from a fixed JSON structure

I have a JSON with dynamic data and not sure how I can retrieve data with JQ.
My JSON is:
{
"RuntimeSources":{
"env-name-DYNAMIC":{
"the-dynamic-value-i-need-to-get":{
"url":""
}
}
},
"DeploymentId":147,
"Serial":158
}
'env-name-DYNAMIC' is dynamic and 'the-dynamic-value-i-need-to-get' is the same.
The json structure is always the same. How can I get 'the-dynamic-value-i-need-to-get'? Also I may need to retrieve 'env-name-DYNAMIC'
Use the keys[] attribute
.RuntimeSources | keys[]
and also
.RuntimeSources | keys[] as $k | .[$k] | keys[]
Since you had also mentioned, the structure doesn't change, you can just select the paths that contains 3 levels
paths | select( length == 3 ) | .[1]
paths | select( length == 3 ) | .[2]
I managed to get it with:
jq '.RuntimeSources | .[] | keys'
Not sure if it's the best solution, but did the trick.

jq: error (at ec-state:1028): Cannot iterate over null (null)

I have a lengthy JSON file and I execute the command to get the output shown below:
jq -s '.[]
| ."lrouter/show"[]
| del( . | select(.type == "TUNNEL-VRF"))
| del(.ports[] | select(.type == "blackhole" or .type == "cpu-port" or .type == "loopback"))
| "Name: \(.name)" ,
"UUID: \(.uuid)" ,
(.ports[] | {Port_Name: .name,
Port_Type: .type,
Port_Peer: .peer,
Port_IPs: .ips[],
Port_Admin_Up: .admin_up,
Port_Op_State: .op_state_up } )' ec-state
"Name: SR-t0-uplink"
"UUID: 23354d26-6994-46d9-b78c-bb565a1c13f2"
{
"Port_Name": "uplink",
"Port_Type": "uplink",
"Port_Peer": "d78089f6-71b5-4c8e-a477-69ee01f17c5c",
"Port_IPs": "1.1.13.5/24",
"Port_Admin_Up": true,
"Port_Op_State": true
}
{
"Port_Name": "bp-sr0-port",
"Port_Type": "backplane",
"Port_Peer": null,
"Port_IPs": "169.254.0.2/28",
"Port_Admin_Up": false,
"Port_Op_State": false
}
jq: error (at ec-state:1028): Cannot iterate over null (null)
I get the desired result however, I also get the jq error at the end of the result. Just curious to know what am I doing incorrectly with the query.
Since your input is large, you might consider adding assertions or equivalent. Since your program evidently expects arrays at various points, you could instrument it with a function such as:
def q($n; $msg):
if type == "array" or type == "object"
then .
else error("\($msg): composite expected # \($n) vs \(.)")
end;
Your program could then be instrumented as follows:
range(0;length) as $n
| .[$n]
| ."lrouter/show" | q($n; 2) | .[]
| del( . | select(.type == "TUNNEL-VRF"))
| del(.ports | q($n; 3) | .[] | select(.type == "blackhole" or .type == "cpu-port" or .type == "loopback"))
| "Name: \(.name)" ,
"UUID: \(.uuid)" ,
(.ports[] | {Port_Name: .name,
Port_Type: .type,
Port_Peer: .peer,
Port_IPs: (.ips | q($n; 4) |.[]),
Port_Admin_Up: .admin_up,
Port_Op_State: .op_state_up } )

jq create output in many separate files

given the following json:
[
{"_id":{"$oid":"6d2"},"jlo":"ΕΙ AJSB","dd":"d5f"},
{"_id":{"$oid":"c6d3"},"jlo":"ΕΙ ALKSB","dd":"5d9"},
{"_id":{"$oid":"b0cc6d4"},"jlo":"ΕΙ AGHTSB","dd":"1b1"},
{"_id":{"$oid":"6d2"},"jlo":"ΕPOWΙ AJSB","dd":"d5f"},
{"_id":{"$oid":"c6d3"},"jlo":"ΕGTΙ ALKSB","dd":"5d9"},
{"_id":{"$oid":"b0cc6d4"},"jlo":"ΕLKΙ AGHTSB","dd":"1b1"}
]
what i need to do is have as output for each discrete value of the ll element, the unique values of ta, in a separate file, named after a one to one representation where each dd code is substituted with a human readable representation:
d5f:departmentone
5d9:departmentalt
1b1:departshort
Desired output, in a per row basis, each unique value of jlo with the count of times it was found in each dd element so we get in the end something like this:
first file named departmentone.txt:
ΕΙ AJSB 1
ΕPOWΙ AJSB 1
second file named departmentalt.txt
ΕΙ ALKSB 1
ΕGTΙ ALKSB 1
third file named departshort.txt
ΕΙ AGHTSB 2
i have tried with map and reduce, group_by, sort_by, with really poor results
Only one invocation of jq is necessary. To allocate the output to the separate files, you can combine this one invocation with a single invocation to awk, or you could use a shell loop as illustrated below.
First, here's an illustration of how the shell pipeline would look:
jq -r --rawfile dd2name dd2name.tsv -f group.jq input.json |
while IFS=$'\t' read -r f v ; do echo "$v" >> "$f" ; done
This assumes that the mapping to filenames is in a TSV file named dd2name.tsv, and that the following jq program is in group.jq:
def dict:
split("\n") | map(select(length>0) | split("\t"))
| INDEX(.[0]) | map_values(.[1]);
($dd2name | dict) as $dict
| ($dict | keys_unsorted[]) as $dd
| map(select(.dd == $dd))
| group_by(.jlo)
| map("\($dict[$dd])\t\(.[0].jlo) \(length)")[]
As the name suggests, the dict function creates a dictionary giving the mapping of .dd values to the filenames. It assumes the availability of INDEX. If your jq does not have INDEX, then now would be an excellent time to upgrade your jq; otherwise, its def can easily be copied from builtin.jq (google: builtin.jq "def INDEX"), or you could replace the last line by: | reduce .[] as $p ({}; .[$p[0]] = $p[1]);
awk-based solution
The following invocation of awk can be used instead of the while ... done command above:
awk -F\\t 'fn && (fn!=$1) {close(fn)}; {fn=$1; print $2 >> fn}'
Season to taste
If the dd2name.tsv mapping file does not contain the ".txt" suffix, it can easily be added in any of a variety of ways, according to taste.
Note also that the proposed solutions above make some assumptions, notably that the .jlo values do not contain tabs, newlines, or NULs. If any of those assumptions is violated, then some tweaking will be required.
I'd do it in three passes, filtering the array with the desired dd and grouping by jlo, then extracting the jlo of the first (guaranteed) item of the array and its length :
map(select(.dd == "d5f")) | group_by(.jlo) | map("\(.[0].jlo) \(length)") | .[]
You can try it here.
Full bash run :
jq --arg dd d5f --raw-output 'map(select(.dd == $dd)) | group_by(.jlo) | map("\(.[0].jlo) \(length)") | .[]' yourJsonFile > departmentone.txt
jq --arg dd 5d9 --raw-output 'map(select(.dd == $dd)) | group_by(.jlo) | map("\(.[0].jlo) \(length)") | .[]' yourJsonFile > departmentalt.txt
jq --arg dd 1b1 --raw-output 'map(select(.dd == $dd)) | group_by(.jlo) | map("\(.[0].jlo) \(length)") | .[]' yourJsonFile > departmentshort.txt
Supposing you have a file named "mapping.txt" with the following content :
d5f:departmentone
5d9:departmentalt
1b1:departshort
You could extract those codes and labels to generate the files :
while IFS=: read -r code label; do
jq --arg dd $code --raw-output 'map(select(.dd == $dd)) | group_by(.jlo) | map("\(.[0].jlo) \(length)") | .[]' yourJsonFile > "$label".txt
done < mapping.txt

How to print out the top-level json after modification of descendants

Hello i managed to create this jq filter .profiles | recurse | .gameDir? | if type == "null" then "" else . end | scan("{REPLACE}.*") | sub("{REPLACE}"; "{REPLACESTRINGHERE}"). it succesfully replaces what i want (checked at jqplay.org) but now i'd like to print the full json and not just the modified strings
Adapting your query:
.profiles |= walk( if type == "object" and has("gameDir")
then .gameDir |=
(if type == "null" then "" else . end
| scan("{REPLACE}.*") | sub("{REPLACE}"; "{REPLACESTRINGHERE}"))
else .
end )
(This can easily be tweaked for greater efficiency.)
If your jq does not have walk, you can google it (jq “def walk”) or snarf its def from the jq FAQ https://github.com/stedolan/jq/wiki/FAQ
walk-free approach
For the record, here's an illustration of a walk-free approach using paths. The following also makes some changes in the computation of the replacement string -- notably it eliminates the use of scan -- so it is not logically equivalent, but is likely to be more useful as well as more efficient.
.profiles |=
( . as $in
| reduce (paths | select(.[-1] == "gameDir")) as $path ($in;
($in | getpath($path)
| if type == "null" then ""
else sub(".*{REPLACE}"; "{REPLACESTRINGHERE}")
end) as $value
| setpath($path; $value) ))

Jq: recursively delete all keys that match a given pattern

How to recursively delete all keys that match a given pattern?
I have following jq config, but it doesn't seem to work:
walk( if (type == "object" and (.[] | test('.*'))) then del(.) else . end)
A robust way (with respect to different jq versions) to delete all keys matching a pattern (say PATTERN) would be to use the idiom:
with_entries(select( .key | test(PATTERN) | not))
Plugging this into walk/1 yields:
walk(if type == "object" then with_entries(select(.key | test(PATTERN) | not)) else . end)