I have the following json file named CMakePresets.json that is a cmake-preset file:
{
"configurePresets": [
{
"name": "default",
"hidden": true,
"generator": "Ninja",
"binaryDir": "${sourceDir}/_build/${presetName}",
"cacheVariables": {
"YIO_DEV": "1",
"BUILD_TESTING": "1"
}
},
{
"name": "debug",
"inherits": "default",
"cacheVariables": {
"CMAKE_BUILD_TYPE": "Debug"
}
},
{
"name": "release",
"inherits": "default",
"binaryDir": "${sourceDir}/_build/Debug",
"cacheVariables": {
"CMAKE_BUILD_TYPE": "Release"
}
},
{
"name": "arm",
"inherits": "debug",
"cacheVariables": {
"CMAKE_TOOLCHAIN_FILE": "${sourceDir}/cmake/Toolchain/arm-none-eabi-gcc.cmake"
}
}
]
}
I want recursively merge with * the configurePresets elements that inherit themselves for a specific entry name. I have example a node with name arm and want to have resulting json object with resolved inheritance. The parent has the name stored inside .inherits of each element. arm inherits over debug which inherits over default.
I could write a bash shell loop that I believe works, with the help of Remove a key:value from an JSON object using jq and this answer:
input=arm
# extract one element
g() { jq --arg name "$1" '.configurePresets[] | select(.name == $name)' CMakePresets.json; };
# get arm element
acc=$(g "$input");
# If .inherits field exists
while i=$(<<<"$acc" jq -r .inherits) && [[ -n "$i" && "$i" != "null" ]]; do
# remove it from input
a=$(<<<"$acc" jq 'del(.inherits)');
# get parent element
b=$(g "$i");
# merge parent with current
acc=$(printf "%s\n" "$b" "$a" | jq -s 'reduce .[] as $item ({}; . * $item)');
done;
echo "$acc"
outputs, which I believe is the expected output for arm:
{
"name": "arm",
"hidden": true,
"generator": "Ninja",
"binaryDir": "${sourceDir}/_build/${presetName}",
"cacheVariables": {
"YIO_DEV": "1",
"BUILD_TESTING": "1",
"CMAKE_BUILD_TYPE": "Debug",
"CMAKE_TOOLCHAIN_FILE": "${sourceDir}/cmake/Toolchain/arm-none-eabi-gcc.cmake"
}
}
But I want to write it in jq. I tried and jq language is not intuitive for me. I can do it for example for two (ie. countable) elements:
< CMakePresets.json jq --arg name "arm" '
def g(n): .configurePresets[] | select(.name == n);
g($name) * (g($name) | .inherits) as $name2 | g($name2)
'
But I do not know how to do reduce .[] as $item ({}; . * $item) when the $item is really g($name) that depends on the last g($name) | .inherits. I tried reading jq manual and learning about variables and loops, but jq has a very different syntax. I tried to use while, but that's just syntax error that I do not understand and do not know how to fix. I guess while and until might not be right here, as they operate on previous loop output, while the elements are always from root.
$ < CMakePresets.json jq --arg name "arm" 'def g(n): .configurePresets[] | select(.name == n);
while(g($name) | .inherits as $name; g($name))
'
jq: error: syntax error, unexpected ';', expecting '|' (Unix shell quoting issues?) at <top-level>, line 2:
while(g($name) | .inherits as $name; g($name))
jq: 1 compile error
How to write such loop in jq language?
Assuming the inheritance hierarchy contains no loops, as is the case with the example, we can break the problem down into the pieces shown below:
# Use an inner function of arity 0 to take advantage of jq's TCO
def inherits_from($dict):
def from:
if .name == "default" then .
else $dict[.inherits] as $next
| ., ($next | from)
end;
from;
def chain($start):
INDEX(.configurePresets[]; .name) as $dict
| $dict[$start] | inherits_from($dict);
reduce chain("arm") as $x (null;
($x.cacheVariables + .cacheVariables) as $cv
| $x + .
| .cacheVariables = $cv)
| del(.inherits)
This produces the desired output efficiently.
One advantage of the above formulation of a solution is that it can easily be modified to handle circular dependencies.
Using recurse/1
inherits_from/1 could also be defined using the built-in function recurse/1:
def inherits_from($dict):
recurse( select(.name != "default") | $dict[.inherits]) ;
or perhaps more interestingly:
def inherits_from($dict):
recurse( select(.inherits) | $dict[.inherits]) ;
Using *
Using * to combine objects has a high overhead because of its recursive semantics, which is often either not required or not wanted. However,
if it is acceptable here to use * for combining the objects, the above can be simplified to:
def inherits_from($dict):
recurse( select(.inherits) | $dict[.inherits]) ;
INDEX(.configurePresets[]; .name) as $dict
| $dict["arm"]
| reduce inherits_from($dict) as $x ({}; $x * .)
| del(.inherits)
Writing a recursive function is actually simple, once you get the hang of it:
jq --arg name "$1" '
def _get_in(input; n):
(input[] | select(.name == n)) |
(if .inherits then .inherits as $n | _get_in(input; $n) else {} end) * .;
def get(name):
.configurePresets as $input | _get_in($input; name);
get($name)
' "$presetfile"
First I filter only .configurePresets then in a function I get input[] | select(.name == n) only the part I am interested in. Then if .inherits if it has inherits, then .inherits as $n | _get_in(input; $n) take the name in inherits and call itself again. Else return else {} end empty. Then that is * . merged with the result of input[] | select(.name == n) - the itself. So it recursively loads all the {} * (input[]|select()) * (input[]|select()) * (input[]|select()).
I pulled some config variables from a json file with jq.
And once modified, i want to write the whole config array (which can contain keys that were not there at first) to the json file.
The "foreach" part seems quite obvious.
But how to express "change keyA by value A or add keyA=>valueA" to the conf file ?
I'm stuck with something like
for key in "${!conf[#]}"; do
value=${conf[$key]}
echo $key $value
jq --arg key $key --arg value $value '.$key = $value' $conf_file > $new_file
done
Thanks
Extended solution with single jq invocation:
Sample conf array:
declare -A conf=([status]="finished" [nextTo]="2018-01-24" [result]=true)
Sample file conf.json:
{
"status": "running",
"minFrom": "2018-01-23",
"maxTo": "2018-01-24",
"nextTo": "2018-01-23",
"nextFrom": "2018-01-22"
}
Processing:
jq --arg data "$(paste -d':' <(printf "%s\n" "${!conf[#]}") <(printf "%s\n" "${conf[#]}"))" \
'. as $conf | map($data | split("\n")[]
| split(":") | {(.[0]) : .[1]})
| add | $conf + .' conf.json > conf.tmp && mv conf.tmp conf.json
The resulting conf.json contents:
{
"status": "finished",
"minFrom": "2018-01-23",
"maxTo": "2018-01-24",
"nextTo": "2018-01-24",
"nextFrom": "2018-01-22",
"result": "true"
}
I finally run into the following solution : not as compact as RomanPerekhrest, but a bit more human-readable. Thanks.
function update_conf() {
echo update_conf
if [ -n "$1" ] && [ -n "$2" ]; then
conf[$1]=$2
fi
for key in "${!conf[#]}"; do
value=${conf[$key]}
jq --arg key "$key" --arg value "$value" '.[$key] = $value' $confFile > $confTempFile && mv $confTempFile $confF$
done
}
Here is my config.json:
{
"env": "dev",
"dev": {
"projects" : {
"prj1": {
"dependencies": {},
"description": ""
}
}
}
}
Here are my bash commands:
PRJNAME='prj1'
echo $PRJNAME
jq --arg v "$PRJNAME" '.dev.projects."$v"' config.json
jq '.dev.projects.prj1' config.json
The output:
prj1
null
{
"dependencies": {},
"description": ""
}
So $PRJNAME is prj1, but the first invocation only outputs null.
Can someone help me?
The jq program .dev.projects."$v" in your example will literally try to find a key named "$v". Try the following instead:
jq --arg v "$PRJNAME" '.dev.projects[$v]' config.json
You can use --argjson too when you make your json.
--arg a v # set variable $a to value <v>;
--argjson a v # set variable $a to JSON value <v>;
As asked in a comment above there's a way to pass multiple argumets.
Maybe there's a more elegant way, but it works.
If you are sure always all keys needed you can use this:
jq --arg key1 $k1 --arg key2 $k2 --arg key3 $k3 --arg key4 $k4 '.[$key1] | .[$key2] | .[$key3] | .[$key4] '
If the key isn't always used you could do it like this:
jq --arg key $k ' if key != "" then .[$key] else . end'
If key sometimes refers to an array:
jq --arg key $k ' if type == "array" then .[$key |tonumber] else .[$key] end'
of course you can combine these!
you can do this:
key="dev.projects.prj1"
filter=".$key"
cat config.json | jq $filter
My example bash command to replace an array in a json file.
Everything is in variables:
a=/opt/terminal/conf/config.json ; \
b='["alchohol"]' ; \
c=terminal.canceledInspections.resultSendReasons ; \
cat $a | jq .$c ; \
cat $a | jq --argjson b $b --arg c $c 'getpath($c / ".") = $b' | sponge $a ; \
cat $a | jq .$c
Need help to parse and print only category values either using jq or jsawk or shell script.
{
"fine_grained": {
"dog": [
{
"category": "cocker spaniel",
"mark": 0.9958831668
}
]
},
"coarse": [
{
"category": "dog",
"mark": 0.948208034
}
]
}
Assuming all category values are simple strings and you want all category values, regardless of where it is in the JSON, you could use this filter using jq:
.. | objects.category // empty
This returns the following strings:
"cocker spaniel"
"dog"
Here is a solution which uses leaf_paths and select to find all the paths with a leaf "category" member and then extract the corresponding values with foreach
foreach (leaf_paths | select(.[-1] == "category")) as $p (
.
; .
; getpath($p)
)
If your input is in a file called input.json and the above filter is in a file called filter.jq then the shell command
jq -f filter.jq input.json
should produce
"cocker spaniel"
"dog"
You can use the -r flag if you don't want the quotes in the output.
EDIT: I now realize a filter of the form foreach E as $X (.; .; R) can almost always be rewritten as E as $X | R so the above is really just
(leaf_paths | select(.[-1] == "category")) as $p
| getpath($p)
Are there any command line utilities that can be used to find if two JSON files are identical with invariance to within-dictionary-key and within-list-element ordering?
Could this be done with jq or some other equivalent tool?
Examples:
These two JSON files are identical
A:
{
"People": ["John", "Bryan"],
"City": "Boston",
"State": "MA"
}
B:
{
"People": ["Bryan", "John"],
"State": "MA",
"City": "Boston"
}
but these two JSON files are different:
A:
{
"People": ["John", "Bryan", "Carla"],
"City": "Boston",
"State": "MA"
}
C:
{
"People": ["Bryan", "John"],
"State": "MA",
"City": "Boston"
}
That would be:
$ some_diff_command A.json B.json
$ some_diff_command A.json C.json
The files are not structurally identical
If your shell supports process substitution (Bash-style follows, see docs):
diff <(jq --sort-keys . A.json) <(jq --sort-keys . B.json)
Objects key order will be ignored, but array order will still matter. It is possible to work-around that, if desired, by sorting array values in some other way, or making them set-like (e.g. ["foo", "bar"] → {"foo": null, "bar": null}; this will also remove duplicates).
Alternatively, substitute diff for some other comparator, e.g. cmp, colordiff, or vimdiff, depending on your needs. If all you want is a yes or no answer, consider using cmp and passing --compact-output to jq to not format the output for a potential small performance increase.
Use jd with the -set option:
No output means no difference.
$ jd -set A.json B.json
Differences are shown as an # path and + or -.
$ jd -set A.json C.json
# ["People",{}]
+ "Carla"
The output diffs can also be used as patch files with the -p option.
$ jd -set -o patch A.json C.json; jd -set -p patch B.json
{"City":"Boston","People":["John","Carla","Bryan"],"State":"MA"}
https://github.com/josephburnett/jd#command-line-usage
Since jq's comparison already compares objects without taking into account key ordering, all that's left is to sort all lists inside the object before comparing them. Assuming your two files are named a.json and b.json, on the latest jq nightly:
jq --argfile a a.json --argfile b b.json -n '($a | (.. | arrays) |= sort) as $a | ($b | (.. | arrays) |= sort) as $b | $a == $b'
This program should return "true" or "false" depending on whether or not the objects are equal using the definition of equality you ask for.
EDIT: The (.. | arrays) |= sort construct doesn't actually work as expected on some edge cases. This GitHub issue explains why and provides some alternatives, such as:
def post_recurse(f): def r: (f | select(. != null) | r), .; r; def post_recurse: post_recurse(.[]?); (post_recurse | arrays) |= sort
Applied to the jq invocation above:
jq --argfile a a.json --argfile b b.json -n 'def post_recurse(f): def r: (f | select(. != null) | r), .; r; def post_recurse: post_recurse(.[]?); ($a | (post_recurse | arrays) |= sort) as $a | ($b | (post_recurse | arrays) |= sort) as $b | $a == $b'
Pulling in the best from the top two answers to get a jq based json diff:
diff \
<(jq -S 'def post_recurse(f): def r: (f | select(. != null) | r), .; r; def post_recurse: post_recurse(.[]?); (. | (post_recurse | arrays) |= sort)' "$original_json") \
<(jq -S 'def post_recurse(f): def r: (f | select(. != null) | r), .; r; def post_recurse: post_recurse(.[]?); (. | (post_recurse | arrays) |= sort)' "$changed_json")
This takes the elegant array sorting solution from https://stackoverflow.com/a/31933234/538507 (which allows us to treat arrays as sets) and the clean bash redirection into diff from https://stackoverflow.com/a/37175540/538507 This addresses the case where you want a diff of two json files and the order of the array contents is not relevant.
Here is a solution using the generic function walk/1:
# Apply f to composite entities recursively, and to atoms
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
def normalize: walk(if type == "array" then sort else . end);
# Test whether the input and argument are equivalent
# in the sense that ordering within lists is immaterial:
def equiv(x): normalize == (x | normalize);
Example:
{"a":[1,2,[3,4]]} | equiv( {"a": [[4,3], 2,1]} )
produces:
true
And wrapped up as a bash script:
#!/bin/bash
JQ=/usr/local/bin/jq
BN=$(basename $0)
function help {
cat <<EOF
Syntax: $0 file1 file2
The two files are assumed each to contain one JSON entity. This
script reports whether the two entities are equivalent in the sense
that their normalized values are equal, where normalization of all
component arrays is achieved by recursively sorting them, innermost first.
This script assumes that the jq of interest is $JQ if it exists and
otherwise that it is on the PATH.
EOF
exit
}
if [ ! -x "$JQ" ] ; then JQ=jq ; fi
function die { echo "$BN: $#" >&2 ; exit 1 ; }
if [ $# != 2 -o "$1" = -h -o "$1" = --help ] ; then help ; exit ; fi
test -f "$1" || die "unable to find $1"
test -f "$2" || die "unable to find $2"
$JQ -r -n --argfile A "$1" --argfile B "$2" -f <(cat<<"EOF"
# Apply f to composite entities recursively, and to atoms
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
def normalize: walk(if type == "array" then sort else . end);
# Test whether the input and argument are equivalent
# in the sense that ordering within lists is immaterial:
def equiv(x): normalize == (x | normalize);
if $A | equiv($B) then empty else "\($A) is not equivalent to \($B)" end
EOF
)
POSTSCRIPT: walk/1 is a built-in in versions of jq > 1.5, and can therefore be omitted if your jq includes it, but there is no harm in including it redundantly in a jq script.
POST-POSTSCRIPT: The builtin version of walk has recently been changed so that it no longer sorts the keys within an object. Specifically, it uses keys_unsorted. For the task at hand, the version using keys should be used.
There's an answer for this here that would be useful.
Essentially you can use the Git diff functionality (even for non-Git tracked files) which also includes colour in the output:
git diff --no-index payload_1.json payload_2.json
One more tool for those to which the previous answers are not a good fit, you can try jdd.
It's HTML based so you can either use it online at www.jsondiff.com or, if you prefer running it locally, just download the project and open the index.html.
Perhaps you could use this sort and diff tool: http://novicelab.org/jsonsortdiff/ which first sorts the objects semantically and then compares it. It is based on https://www.npmjs.com/package/jsonabc
If you also want to see the differences, using #Erik's answer as inspiration and js-beautify:
$ echo '[{"name": "John", "age": 56}, {"name": "Mary", "age": 67}]' > file1.json
$ echo '[{"age": 56, "name": "John"}, {"name": "Mary", "age": 61}]' > file2.json
$ diff -u --color \
<(jq -cS . file1.json | js-beautify -f -) \
<(jq -cS . file2.json | js-beautify -f -)
--- /dev/fd/63 2016-10-18 13:03:59.397451598 +0200
+++ /dev/fd/62 2016-10-18 13:03:59.397451598 +0200
## -2,6 +2,6 ##
"age": 56,
"name": "John Smith"
}, {
- "age": 67,
+ "age": 61,
"name": "Mary Stuart"
}]
In JSONiq, you can simply use the deep-equal function:
deep-equal(
{
"People": ["John", "Bryan", "Carla"],
"City": "Boston",
"State": "MA"
},
{
"People": ["Bryan", "John"],
"State": "MA",
"City": "Boston"
}
)
which returns
false
You can also read from files (locally or an HTTP URL also works) like so:
deep-equal(
json-doc("path to doc A.json"),
json-doc("path to doc B.json")
)
A possible implementation is RumbleDB.
However, you need to be aware that it is not quite correct that the first two documents are the same: JSON defines arrays as ordered lists of values.
["Bryan", "John"]
is not the same as:
["John", "Bryan"]