verify that a json field exists with jq and bash? - json

I have a script that uses jq for parsing a json string MESSAGE (that is read from another application). Meanwhile the json has changed and a field is split in 2 fields: file_path is now split into folder and file. The script was reading the file_path, now the folder may not be present, so for creating the path of the file I have to verify if the field is there. I have search for a while on the internet, and manage to do:
echo $(echo $MESSAGE | jq .folder -r)$'/'$(echo $MESSAGE | jq .file -r)
if [ $MESSAGE | jq 'has(".folder")' -r ]
then
echo $(echo $MESSAGE | jq .folder -r)$'/'$(echo $MESSAGE | jq .file -r)
else
echo $(echo $MESSAGE | jq .file -r)
fi
where MESSAGE='{"folder":"FLDR","file":"fl"}' or MESSAGE='{"file":"fl"}'
The first line is printing FLDR/fl or null/fl if the folder field is not present. So I have thought to create an if that is verifying if the folder field is present or not, but it seems that I am doing it wrong and cannot figure out what is wrong. The output is
bash: [: missing `]'
jq: ]: No such file or directory
null/fl

I'd do the whole thing in a jq filter:
echo "$MESSAGE" | jq -r '[ .folder, .file ] | join("/")'
In the event that you want to do it with bash (or to learn how to do this sort of thing in bash), two points:
Shell variables should almost always be quoted when they are used (i.e., "$MESSAGE" instead of $MESSAGE). You will run into funny problems if one of the strings in your JSON ever contains a shell metacharacter (such as *) and you forgot to do that; the string will be subject to shell expansion (and that * will be expanded into a list of files in the current working directory).
A shell if accepts as condition a command, and the decision where to branch is made depending on the exit status of that command (true if the exit status is 0, false otherwise). The [ you attempted to use is just a command (an alias for test, see man test) and not special in any way.
So, the goal is to construct a command that exits with 0 if the JSON object has a folder property, non-zero otherwise. jq has a -e option that makes it return 0 if the last output value was not false or null and non-zero otherwise, so we can write
if echo "$MESSAGE" | jq -e 'has("folder")' > /dev/null; then
echo "$MESSAGE" | jq -r '.folder + "/" + .file'
else
echo "$MESSAGE" | jq -r .file
fi
The > /dev/null bit redirects the output from jq to /dev/null (where it is ignored) so that we don't see it on the console.

Related

JSON jq execute with Process Substitution

I create bash script to parse json file and generate hosts. For that I use jq but I cannot get it work with variable domain_count changing.
domain_count=0
jq -r .domains[] variables.json | while read domain; do
host="0.0.0.0 ${domain}"
echo $host;
# ((domain_count++))
done
echo $domain_count
It is still 0.
So that is because Process Substitution. I tried change it different ways. But non of it works.
while read domain
do
host="0.0.0.0 ${domain}"
echo $host;
((domain_count++))
done < <(jq -r .domains[] variables.json)
echo $domain_count
I got next error
generate.sh: line 20: syntax error near unexpected token `<'
generate.sh: line 20: `done < <(jq -r .domains[] variables.json)'
Different steps of a pipeline behave like sub shells. Variables are inherited from the parent, but changes do not propagate back to the parent. You have to make your echo statement part of this step in the pipeline:
domain_count=0
jq -r '.domains[]' variables.json | {
while read -r domain; do
host="0.0.0.0 ${domain}"
echo "$host";
domain_count+$((domain_count++))
done
echo $domain_count
}
But if all you want to do is count the number of domains/hosts, that can be done with jq directly:
domain_count=$(jq '.domains|length' variables.json)
And to output all domains formatted as hosts:
jq -r '.domains[] | "0.0.0.0 \(.)"' variables.json
Summarized, without losing functionality, your script can be shortened to:
jq -r '.domains[] | "0.0.0.0 \(.)"' variables.json
domain_count=$(jq '.domains|length' variables.json)

Parse multiple json files and output the match/hits against the regex with associated file names

Currently, the cat command piped to jq helps me to parse multiple JSON files in my working directory and screen against the regex pattern matching email ids available in all in the files. However, am keen to identify the file name also in which the regex pattern is being hit/matched
cat *.json | jq '. as $data | [path(..| select(scalars and (tostring | test("^[a-zA-Z0-9+_.-]+#[a-zA-Z0-9.-]+$", "ixn")))) ] | map({ (.|join(".")): (. as $path | .=$data | getpath($path)) }) | reduce .[] as $item ({}; . * $item)'
Request your kind help tweaking the command to print $filename. thanks!
input_filename evaluates to the input file name of the file currently being read (after it has been opened). For STDIN, it evaluates to "<stdin>":
jq 'input_filename, input_filename' <<< 1
"<stdin>"
"<stdin>"
It works with the -n command-line option, but only after an input or inputs function has been called:
jq -n 'input_filename, (input | input_filename)' <<< 1
null
"<stdin>"
For a jq-internal solution use input_filename as #peak suggested. Here's an external solution which iterates over your input files and passes the file name as variable into jq. This approach, however, calls jq once for each input file (as opposed to your cat *.json | jq ... approach which has just one call), so you might run into performance issues when applied to a larger number of input files.
for f in *.json
do jq --arg f "$f" '. as $data | ... (use $f here) ...' "$f"
done

jq truncates ENV variable after whitespace

Trying to write a bash script that replaces values in a JSON file we are running into issues with Environment Variables that contain whitespaces.
Given an original JSON file.
{
"version": "base",
"myValue": "to be changed",
"channelId": 0
}
We want to run a command to update some variables in it, so that after we run:
CHANNEL_ID=1701 MY_VALUE="new value" ./test.sh
The JSON should look like this:
{
"version": "base",
"myValue": "new value",
"channelId": 1701
}
Our script is currently at something like this:
#!/bin/sh
echo $MY_VALUE
echo $CHANNEL_ID
function replaceValue {
if [ -z $2 ]; then echo "Skipping $1"; else jq --argjson newValue \"${2}\" '. | ."'${1}'" = $newValue' build/config.json > tmp.json && mv tmp.json build/config.json; fi
}
replaceValue channelId ${CHANNEL_ID}
replaceValue myValue ${MY_VALUE}
In the above all values are replaced by string and strings are getting truncated at whitespace. We keep alternating between this issue and a version of the code where substitutions just stop working entirely.
This is surely an issue with expansions but we would love to figure out, how we can:
- Replace values in the JSON with both strings and values.
- Use whitespaces in the strings we pass to our script.
You don't have to mess with --arg or --argjson to import the environment variables into jq's context. It can very well read the environment on its own. You don't need a script separately, just set the values along with the invocation of jq
CHANNEL_ID=1701 MY_VALUE="new value" \
jq '{"version": "base", myValue: env.MY_VALUE, channelId: env.CHANNEL_ID}' build/config.json
Note that in the case above, the variables need not be exported globally but just locally to the jq command. This allows you to not export multiple variables into the shell and pollute the environment, but just the ones needed for jq to construct the desired JSON.
To make the changes back to the original file, do > tmp.json && mv tmp.json build/config.json or more clearly download the sponge(1) utility from moreutils package. If present, you can pipe the output of jq as
| sponge build/config.json
Pass variables with --arg. Do:
jq --arg key "$1" --arg value "$2" '.[$key] = $value'
Notes:
#!/bin/sh indicates that this is posix shell script, not bash. Use #!/bin/bash in bash scripts.
function replaceValue { is something from ksh shell. Prefer replaceValue() { to declare functions. Bash obsolete and deprecated syntax.
Use newlines in your script to make it readable.
--argjson passes a json formatted argument, not a string. Use --arg for that.
\"${2}\" doesn't quote $2 expansion - it only appends and suffixes the string with ". Because the expansion is not qouted, word splitting is performed, which causes your input to be split on whitespaces when creating arguments for jq.
Remember to quote variable expansions.
Use http://shellcheck.net to check your scripts.
. | means nothing in jq, it's like echo $(echo $(echo))). You could jq '. | . | . | . | . | .' do it infinite number of times - it passes the same thing. Just write the thing you want to do.
Do:
#!/bin/bash
echo "$MY_VALUE"
echo "$CHANNEL_ID"
replaceValue() {
if [ -z "$2" ]; then
echo "Skipping $1"
else
jq --arg key "$1" --arg value "$2" '.[$key] = $value' build/config.json > tmp.json &&
mv tmp.json build/config.json
fi
}
replaceValue channelId "${CHANNEL_ID}"
replaceValue myValue "${MY_VALUE}"
#edit Replaced ."\($key)" with easier .[$key]
jq allows you to build new objects:
MY_VALUE=foo;
CHANNEL_ID=4
echo '{
"version": "base",
"myValue": "to be changed",
"channelId": 0
}' | jq ". | {\"version\": .version, \"myValue\": \"$MY_VALUE\", \"channelId\": $CHANNEL_ID}"
The . selects the whole input, and inputs that (|) to the construction of a new object (marked by {}). For version is selects .version from the input, but you can set your own values for the other two. We use double quotes to allow the Bash variable expansion, which means escaping the double quotes in the JSON.
You'll need to adapt my snippet above to scriptify it.

Exporting JSON to environment variables

If I have a JSON like this,
{
"hello1": "world1",
"testk": "testv"
}
And I want to export each of these key-value pairs as environment variables, how to do it via shell script? So for example, when I write on the terminal, echo $hello1, world1 should be printed and similarly for other key-value pairs?
Note: The above JSON is present in a variable called $values and not in a file.
I know it will be done via jq and written a shell script for this, but it doesn't work.
for row in $(echo "${values}" | jq -r '.[]'); do
-jq() {
echo ${row} | jq -r ${1}
}
echo $(_jq '.samplekey')
done
Borrowing from this answer which does all of the hard work of turning the JSON into key=value pairs, you could get these into the environment by looping over the jq output and exporting them:
for s in $(echo $values | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" ); do
export $s
done
If the variables being loaded contain embedded whitespace, this is also reasonable, if slightly more complex:
while read -rd $'' line
do
export "$line"
done < <(jq -r <<<"$values" \
'to_entries|map("\(.key)=\(.value)\u0000")[]')
Using command substitution $() :
# $(jq -r 'keys[] as $k | "export \($k)=\(.[$k])"' file.json)
# echo $testk
testv
Edit : Responding to this comment
You should do
$( echo "$values" | jq -r 'keys[] as $k | "export \($k)=\(.[$k])"' )
Just mind the double quotes around $values
Note: Couldn't confirm if there is security implication to this approach, that is if the user could manipulate the json to wreak havoc.
Another way, without using jq, is to parse the json with grep & sed:
for keyval in $(grep -E '": [^\{]' my.json | sed -e 's/: /=/' -e "s/\(\,\)$//"); do
echo "export $keyval"
eval export $keyval
done
Explanation:
First, grep will filter all "key" : value pairs (value can be
"string", number, or boolean).
Then, sed will replace : with =, and remove trailing ,.
Lastly, exporting the "key"=value with eval
Here's an output example, exporting json keys, from an AWS record-set:
export "Name"="\052.apps.nmanos-cluster-a.devcluster.openshift.com."
export "Type"="A"
export "HostedZoneId"="Z67SXBLZRQ7X7T"
export "DNSName"="a24070461d50270e-1391692.us-east-1.elb.amazonaws.com."
export "EvaluateTargetHealth"=false
None of the existing answers preserve whitespace in the values in a POSIX shell. The following line will use jq to take each key:value of some JSON and export them as environment variables, properly escaping whitespace and special characters.
2023-01-28: BUGFIX UPDATE:
My previous answer did not work for all possible values and could cause errors. Please instead use the following line, which uses jq's #sh format string to properly escape values for the shell. You must also enclose everything after eval in quotes to preserve newlines. I've updated the sample JSON file to include more characters to test with.
This answer now appears to be the only one that handles all cases. There are no loops and it's one line to export all values. The downside is that it uses eval, which is theoretically dangerous... but because the entire key=value is now being escaped for the shell, this should be safe to use.
New answer (use this one):
eval "export $(echo "$values" \
| jq -r 'to_entries | map("\(.key)=\(.value)") | #sh')"
Old answer (don't use this one):
eval export $(echo "$values" \
| jq -r 'to_entries|map("\"\(.key)=\(.value|tostring)\"")|.[]' )
edit thanks #Delthas for pointing out a missing 'export'
Sample JSON file:
bash-5.2$ cat <<'EOJSON' > foo.json
{
"foo_1": "bar 1",
"foo_2": "This ! is ' some # weird $text { to ( escape \" here",
"foo_3": "this is some \nsample new line\n text to\ntry and escape"
}
EOJSON
Sample script:
bash-5.2$ cat <<'EOSH' > foo.sh
values="`cat foo.json`"
eval "export $(echo "$values" | jq -r 'to_entries | map("\(.key)=\(.value)") | #sh')"
export
echo "foo_2: $foo_2"
echo "foo_3: $foo_3"
EOSH
Running the sample script:
bash-5.2$ env -i sh foo.sh
export PWD='/path/to/my/home'
export SHLVL='1'
export foo_1='bar 1'
export foo_2='This ! is '"'"' some # weird $text { to ( escape " here'
export foo_3='this is some
sample new line
text to
try and escape'
foo_2: This ! is ' some # weird $text { to ( escape " here
foo_3: this is some
sample new line
text to
try and escape
Pros:
no need for Bash
preserves whitespace in values
no loops
(update) properly escapes all values for use in the shell
Cons:
uses eval, which is considered "unsafe". however, because jq is escaping all input, this is unlikely to cause a security issue (unless jq is found to have a bug which does not properly escape data using the #sh filter).
The approach illustrated by the following shell script avoids most (but not all) problems with special characters:
#!/bin/bash
function json2keyvalue {
cat<<EOF | jq -r 'to_entries|map("\(.key)\t\(.value|tostring)")[]'
{
"hello1": "world1",
"testk": "testv"
}
EOF
}
while IFS=$'\t' read -r key value
do
export "$key"="$value"
done < <(json2keyvalue)
echo hello1="$hello1"
echo testk="$testk"
Note that the above assumes that there are no tabs in the keys themselves.
jtc solution:
export $(<file.json jtc -w'[:]<>a:<L>k' -qqT'"{L}={}"')
I've come up with a solution (here in bash):
function source_json_as_environ() {
eval "$(jq -r '
def replace_dot:
. | gsub("\\."; "_");
def trim_spaces:
. | gsub("^[ \t]+|[ \t]+$"; "");
to_entries|map(
"export \(.key|trim_spaces|replace_dot)="
+ "\(.value|tostring|trim_spaces|#sh)"
)|.[]' $#)"
}
And you can use it like this:
$ source_json_as_environ values.json

Using jq to combine json files, getting file list length too long error

Using jq to concat json files in a directory.
The directory contains a few hundred thousand files.
jq -s '.' *.json > output.json
returns an error that the file list is too long. Is there a way to write this that uses a method that will take in more files?
If jq -s . *.json > output.json produces "argument list too long"; you could fix it using zargs in zsh:
$ zargs *.json -- cat | jq -s . > output.json
That you could emulate using find as shown in #chepner's answer:
$ find -maxdepth 1 -name \*.json -exec cat {} + | jq -s . > output.json
"Data in jq is represented as streams of JSON values ... This is a cat-friendly format - you can just join two JSON streams together and get a valid JSON stream.":
$ echo '{"a":1}{"b":2}' | jq -s .
[
{
"a": 1
},
{
"b": 2
}
]
The problem is that the length of a command line is limited, and *.json produces too many argument for one command line. One workaround is to expand the pattern in a for loop, which does not have the same limits as a command line, because bash can iterate over the result internally rather than having to construct an argument list for an external command:
for f in *.json; do
cat "$f"
done | jq -s '.' > output.json
This is rather inefficient, though, since it requires running cat once for each file. A more efficient solution is to use find to call cat with as many files as possible each time.
find . -name '*.json' -exec cat '{}' + | jq -s '.' > output.json
(You may be able to simply use
find . -name '*.json' -exec jq -s '{}' + > output.json
as well; it may depend on what is in the files and how multiple calls to jq using the -s option compares to a single call.)
[EDITED to use find]
One obvious thing to consider would be to process one file at a time, and then "slurp" them:
$ while IFS= read -r f ; cat "$f" ; done <(find . -maxdepth 1 -name "*.json") | jq -s .
This however would presumably require a lot of memory. Thus the following may be closer to what you need:
#!/bin/bash
# "slurp" a bunch of files
# Requires a version of jq with 'inputs'.
echo "["
while read f
do
jq -nr 'inputs | (., ",")' $f
done < <(find . -maxdepth 1 -name "*.json") | sed '$d'
echo "]"