for region in $(jq '.data | keys | .[]' <<< "$data"); do
value=$(jq -r ".data[$region]" <<< "$data");
deliveryRegionId=$(jq -r '.deliveryRegionId' <<< "$value");
json_template='{}';
json_data=$(jq --argjson deliveryRegionId "$deliveryRegionId" --arg deliverableDistance 5000 '.deliveryRegionId=$deliveryRegionId | .deliverableDistance=5000' <<<"$json_template"); echo $json_data;
requestArray=$(jq '. += [$json_data]' <<< $requestArray)
done;
As in the code above, I'm going to create a json value called json_data and add it to the array.
What should I do to make this work?
jq: error: $json_data is not defined at <top-level>, line 1:
. += [$json_data]
jq: 1 compile error
this is error
There's no jq variable named $json_data.
There is a shell variable named that, but you can't access another program's variables.
Provide the value via the environment
json_data="$json_data" jq '. += [ env.json_data ]' <<<"$requestArray"
Provide the value via the environment
export json_data
jq '. += [ env.json_data ]' <<<"$requestArray"
Provide the value as an argument
jq --arg json_data "$json_data" '. += [ $json_data ]' <<<"$requestArray"
There's no reason to use jq so many times! Your entire program can be replaced with this:
requestArray="$(
jq '.data | map( { deliveryRegionId, deliverableDistance: 5000 } )' \
<<<"$data"
)"
Demo on jqplay
Related
I'm looking for a solution to add a new attribute with a JSON object value into an existing JSON file.
My current script:
if [ ! -f "$src_file" ]; then
echo "Source file $src_file does not exists"
exit 1
fi
if [ ! -f "$dst_file" ]; then
echo "Destination file $dst_file does not exists"
exit 1
fi
if ! jq '.devDependencies' "$src_file" >/dev/null 2>&1; then
echo "The key "devDependencies" does not exists into source file $src_file"
exit 1
fi
dev_dependencies=$(jq '.devDependencies' "$src_file" | xargs )
# Extract data from source file
data=$(cat $src_file)
# Add new key-value
data=$(echo $data | jq --arg key "devDependencies" --arg value "$dev_dependencies" '. + {($key): ($value)}')
# Write data into destination file
echo $data > $dst_file
It's working but the devDependencies value from $dev_dependencies is wrote as string:
"devDependencies": "{ #nrwl/esbuild: 15.6.3, #nrwl/eslint-pl[...]".
How can I write it as raw JSON ?
I think you want the --argjson option instead of --arg. Compare
$ jq --arg k '{"foo": "bar"}' -n '{x: $k}'
{
"x": "{\"foo\": \"bar\"}"
}
with
$ jq --argjson k '{"foo": "bar"}' -n '{x: $k}'
{
"x": {
"foo": "bar"
}
}
--arg will create a string variable. Use --argjson to parse the value as JSON (can be object, array or number).
From the docs:
--arg name value:
This option passes a value to the jq program as a predefined variable.
If you run jq with --arg foo bar, then $foo is available in the
program and has the value "bar". Note that value will be treated as a
string, so --arg foo 123 will bind $foo to "123".
Named arguments are also available to the jq program as $ARGS.named.
--argjson name JSON-text:
This option passes a JSON-encoded value to the jq program as a
predefined variable. If you run jq with --argjson foo 123, then $foo
is available in the program and has the value 123.
Note that you don't need multiple invocations of jq, xargs, command substitution or variables (don't forget to quote all your variables when expanding).
To "merge" the contents of two files, read both files with jq and let jq do the work. This avoids all the complications that arise from jumping between jq and shell context. A single line is all that's needed:
jq --slurpfile deps "$dep_file" '. + { devDependencies: $deps[0].devDependencies }' "$source_file" > "$dest_file"
or
jq --slurpfile deps "$dep_file" '. + ($deps[0]|{devDependencies})' "$source_file" > "$dest_file"
alternatively (still a one-liner):
jq --slurpfile deps "$dev_file" '.devDependencies = $deps[0].devDependencies' "$source_file" > "$dest_file"
peak's answer here reminded me of the very useful input filter, which can make the program even shorter as it avoids the variable:
jq '. + (input|{devDependencies})' "$source_file" "$dep_file" > "$dest_file"
I can create a json object with an array using jq like this
echo '{"input":{"names":[]}}' | jq --arg val "alice" '.input.names[0] += $val'| jq --arg val "bob" '.input.names[1] += $val'
which gives
{
"input": {
"names": [
"alice",
"bob"
]
}
}
Now I want to recreate this in a shell script where "names" come from a shell array
#!/bin/sh
names=( alice bob )
ITER=0
for i in "${names[#]}"
do
echo '{"input":{"names":[]}}' | jq --arg val "$i" '.input.names[$ITER] += $val'
echo ${I} ${ITER}
ITER=$(expr $ITER + 1)
done
but I run into
jq: error: $ITER is not defined at <top-level>, line 1:
.input.names[$ITER] += $val
jq: 1 compile error
0
jq: error: $ITER is not defined at <top-level>, line 1:
.input.names[$ITER] += $val
jq: 1 compile error
1
You could get rid of the shell loop and use one jq call:
names=( alice bob )
jq -n --arg names "${names[*]}" '{"input": {"names": ($names / " ") } }'
or
names=( alice bob )
printf '%s\0' "${names[#]}" |
jq -sR '(. / "\u0000") as $names | { "input": { "names": $names } }'
You don't capture the output of an iteration step, so you lose the output from the jq call and start all over again in the next iteration step.
Here's a bash solution which captures the output using $(...) and provides it for the next using <<<. (Note that there are better ways to solve this problem, e.g. without looping but by proving jq all elements to be added at once.)
json='{"input":{"names":[]}}'
for i in alice bob
do json="$(jq --arg val "$i" '.input.names += [$val]' <<< "$json")"
done
echo "$json"
{
"input": {
"names": [
"alice",
"bob"
]
}
}
I have a .json file which I want to read, take all the contents.. and append it to a json string that's in a bash variable
input.json
[{
"maven": {
"coordinates": "somelib",
"repo": "somerepo"
}
},
{
"maven": {
"coordinates": "anotherlib",
"exclusions": ["exclude:this", "*:and-all-that"]
}
}]
OUR_BIG_JSON variable
{
"name": "",
"myarray": [
{
"generic": {
"package": "somepymodule"
}
},
{
"rcann": {
"package": "anothermodule==3.0.0"
}
}
],
and a json I want to append to.. it's residing in a variable so we must echo it as we use jq
command attempt
echo "starting..."
inputs_json=`cat input.json`
echo "$inputs_json"
echo "$OUR_BIG_JSON"
OUR_BIG_JSON=$(echo "${OUR_BIG_JSON}" |./jq '.myarray += [{"date":"today"}]') # This worked..
next line fails
OUR_BIG_JSON=$(echo "${OUR_BIG_JSON}" |./jq '.myarray += ' $inputs_json)
out comes errors when trying to take the $inputs_json contents and just pipe it into the variable.. basically append to the JSON in memory..
jq: error: syntax error, unexpected INVALID_CHARACTER (Windows cmd shell quoting issues?) at <top-level>, line 1: .myarray += \"$inputs_json\" jq: 1 compile error
Using the --argjson option to read in the contents of your shell variable enables you to use it as variable inside the jq filter, too. Your input file can normally be read in via parameter.
jq --argjson v "${OUR_BIG_JSON}" '…your filter here using $v…' input.json
For example:
jq --argjson v "${OUR_BIG_JSON}" '$v + {myarray: ($v.myarray + .)}' input.json
Or by making the input too a variable inside the jq filter
jq --argjson v "${OUR_BIG_JSON}" '. as $in | $v | .myarray += $in' input.json
With the -n option you could also reference the input using input:
jq --argjson v "${OUR_BIG_JSON}" -n '$v | .myarray += input' input.json
And there's also the options --argfile and --slurpfile to read in the input file as variable directly...
As you have tagged bash you could also do it the other way around and use a herestring <<< to read in the bash variable as input and then using either --argfile or --slurpfile to reference the file content through a variable. For instance:
jq --argfile in input.json '.myarray += $in' <<< "${OUR_BIG_JSON}"
I parsed a json file with jq like this :
# cat test.json | jq '.logs' | jq '.[]' | jq '._id' | jq -s
It returns an array like this : [34,235,436,546,.....]
Using bash script i described an array :
# declare -a msgIds = ...
This array uses () instead of [] so when I pass the array given above to this array it won't work.
([324,32,45..]) this causes problem. If i remove the jq -s, an array forms with only 1 member in it.
Is there a way to solve this issue?
We can solve this problem by two ways. They are:
Input string:
// test.json
{
"keys": ["key1","key2","key3"]
}
Approach 1:
1) Use jq -r (output raw strings, not JSON texts) .
KEYS=$(jq -r '.keys' test.json)
echo $KEYS
# Output: [ "key1", "key2", "key3" ]
2) Use #sh (Converts input string to a series of space-separated strings). It removes square brackets[], comma(,) from the string.
KEYS=$(<test.json jq -r '.keys | #sh')
echo $KEYS
# Output: 'key1' 'key2' 'key3'
3) Using tr to remove single quotes from the string output. To delete specific characters use the -d option in tr.
KEYS=$((<test.json jq -r '.keys | #sh')| tr -d \')
echo $KEYS
# Output: key1 key2 key3
4) We can convert the comma-separated string to the array by placing our string output in a round bracket().
It also called compound Assignment, where we declare the array with a bunch of values.
ARRAYNAME=(value1 value2 .... valueN)
#!/bin/bash
KEYS=($((<test.json jq -r '.keys | #sh') | tr -d \'\"))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
Approach 2:
1) Use jq -r to get the string output, then use tr to delete characters like square brackets, double quotes and comma.
#!/bin/bash
KEYS=$(jq -r '.keys' test.json | tr -d '[],"')
echo $KEYS
# Output: key1 key2 key3
2) Then we can convert the comma-separated string to the array by placing our string output in a round bracket().
#!/bin/bash
KEYS=($(jq -r '.keys' test.json | tr -d '[]," '))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
To correctly parse values that have spaces, newlines (or any other arbitrary characters) just use jq's #sh filter and bash's declare -a. (No need for a while read loop or any other pre-processing)
// foo.json
{"data": ["A B", "C'D", ""]}
str=$(jq -r '.data | #sh' foo.json)
declare -a arr="($str)" # must be quoted like this
$ declare -p arr
declare -a arr=([0]="A B" [1]="C'D" [2]="")
The reason that this works correctly is that #sh will produce a space-separated list of shell-quoted words:
$ echo "$str"
'A B' 'C'\''D' ''
and this is exactly the format that declare expects for an array definition.
Use jq -r to output a string "raw", without JSON formatting, and use the #sh formatter to format your results as a string for shell consumption. Per the jq docs:
#sh:
The input is escaped suitable for use in a command-line for a POSIX shell. If the input is an array, the output will be a series of space-separated strings.
So can do e.g.
msgids=($(<test.json jq -r '.logs[]._id | #sh'))
and get the result you want.
From the jq FAQ (https://github.com/stedolan/jq/wiki/FAQ):
𝑸: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
A: One option would be to use mapfile (aka readarray), for example:
mapfile -t array <<< $(jq -c '.[]' input.json)
An alternative that might be indicative of what to do in other shells is to use read -r within a while loop. The following bash script populates an array, x, with JSON texts. The key points are the use of the -c option, and the use of the bash idiom while read -r value; do ... done < <(jq .......):
#!/bin/bash
x=()
while read -r value
do
x+=("$value")
done < <(jq -c '.[]' input.json)
++ To resolve this, we can use a very simple approach:
++ Since I am not aware of you input file, I am creating a file input.json with the following contents:
input.json:
{
"keys": ["key1","key2","key3"]
}
++ Use jq to get the value from the above file input.json:
Command: cat input.json | jq -r '.keys | #sh'
Output: 'key1' 'key2' 'key3'
Explanation: | #sh removes [ and "
++ To remove ' ' as well we use tr
command: cat input.json | jq -r '.keys | #sh' | tr -d \'
Explanation: use tr delete -d to remove '
++ To store this in a bash array we use () with `` and print it:
command:
KEYS=(`cat input.json | jq -r '.keys | #sh' | tr -d \'`)
To print all the entries of the array: echo "${KEYS[*]}"
As a follow-up to Flatten Arbitrary JSON, I'm looking to take the flattened results and make them suitable for doing queries and updates back to the original JSON file.
Motivation: I'm writing Bash (4.2+) scripts (on CentOS 7) that read JSON into a Bash associative array using the JSON selector/filter as the key. I do processing on the associative arrays, and in the end I want to update the JSON with those changes.
The preceding solution gets me close to this goal. I think there are two things that it doesn't do:
It doesn't quote keys that require quoting. For example, the key com.acme would need to be quoted because it contains a special character.
Array indexes are not represented in a form that can be used to query the original JSON.
Existing Solution
The solution from the above is:
$ jq --stream -n --arg delim '.' 'reduce (inputs|select(length==2)) as $i ({};
[$i[0][]|tostring] as $path_as_strings
| ($path_as_strings|join($delim)) as $key
| $i[1] as $value
| .[$key] = $value
)' input.json
For example, if input.json contains:
{
"a.b":
[
"value"
]
}
then the output is:
{
"a.b.0": "value"
}
What is Really Wanted
An improvement would have been:
{
"\"a.b\"[0]": "value"
}
But what I really want is output formatted so that it could be sourced directly in a Bash program (implying the array name is passed to jq as an argument):
ArrayName['"a.b"[0]']='value' # Note 'value' might need escapes for Bash
I'm looking to have the more human-readable syntax above as opposed to the more general:
ArrayName['.["a.b"][0]']='value'
I don't know if jq can handle all of this. My present solution is to take the output from the preceding solution and to post-process it to the form that I want. Here's the work in process:
#!/bin/bash
Flatten()
{
local -r OPTIONS=$(getopt -o d:m:f: -l "delimiter:,mapname:,file:" -n "${FUNCNAME[0]}" -- "$#")
eval set -- "$OPTIONS"
local Delimiter='.' MapName=map File=
while true ; do
case "$1" in
-d|--delimiter) Delimiter="$2"; shift 2;;
-m|--mapname) MapName="$2"; shift 2;;
-f|--file) File="$2"; shift 2;;
--) shift; break;;
esac
done
local -a Array=()
readarray -t Array <<<"$(
jq -c -S --stream -n --arg delim "$Delimiter" 'reduce (inputs|select(length==2)) as $i ({}; .[[$i[0][]|tostring]|join($delim)] = $i[1])' <<<"$(sed 's|^\s*[#%].*||' "$File")" |
jq -c "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" |
sed -e 's|^"||' -e 's|"$||' -e 's|=|\t|')"
if [[ ! -v $MapName ]]; then
local -gA $MapName
fi
. <(
IFS=$'\t'
while read -r Key Value; do
printf "$MapName[\"%s\"]=%q\n" "$Key" "$Value"
done <<<"$(printf "%s\n" "${Array[#]}")"
)
}
declare -A Map
Flatten -m Map -f "$1"
declare -p Map
With the output:
$ ./Flatten.sh <(echo '{"a.b":["value"]}')
declare -A Map='([a.b.0]="value" )'
1) jq is Turing complete, so it's all just a question of which hammer to use.
2)
An improvement would have been:
{
"\"a.b\"[0]": "value"
}
That is easily accomplished using a helper function along these lines:
def flattenPath(delim):
reduce .[] as $s ("";
if $s|type == "number"
then ((if . == "" then "." else . end) + "[\($s)]")
else . + ($s | tostring | if index(delim) then "\"\(.)\"" else . end)
end );
3)
I do processing on the associative arrays, and in the end I want to update the JSON with those changes.
This suggests you might have posed an xy-problem. However, if you really do want to serialize and unserialize some JSON text, then the natural way to do so using jq is using leaf_paths, as illustrated by the following serialization/deserialization functions:
# Emit (path, value) pairs
# Usage: jq -c -f serialize.jq input.json > serialized.json
def serialize: leaf_paths as $p | ($p, getpath($p));
# Usage: jq -n -f unserialize.jq serialized.json
def unserialize:
def pairwise(s):
foreach s as $i ([];
if length == 1 then . + [$i] else [$i] end;
select(length == 2));
reduce pairwise(inputs) as $p (null; setpath($p[0]; $p[1]));
If using bash, you could use readarray (mapfile) to read the paths and values into a single array, or if you want to distinguish between the paths and values more easily, you could (for example) use the approach illustrated by the following:
i=0
while read -r line ; do
path[$i]="$line"; read -r line; value[$i]="$line"
i=$((i + 1))
done < serialized.json
But there are many other alternatives.