parsing json to check whether a field is blank in bash - json

So, lets say, I am trying to write a shell script which does the following:
1) Ping http://localhost:8088/query?key=q1
2) It returns a json response :
{
"status": "success",
"result": []
"query": "q1"
}
or
{
"status": "success",
"result": ["foo"],
"artist_key": "q1"
}
The "result" is either an empty array or filled array..
I am trying to write a shell script which is checking whether "result" is empty list or not?

Something like this would work:
# Assume result is from curl, but could be in a file or whatever
if curl "http://localhost:8088/query?key=q1" | grep -Pq '"result":\s+\[".+"\]'; then
echo "has result"
else
echo "does not have result"
fi
However, I'm assuming these are on separate lines. If not, there are linters for format it.
Edited (based on the jq comment), here's a jq solution as suggested by Adrian Frühwirth:
result=$( curl "http://localhost:8088/query?key=q1" | jq '.result' )
if [ "$result" == "[]" ]; then
echo "does not have result"
else
echo "has result"
fi
I have learned something new today. And as I play around with this more, maybe it's better to do this:
result=$( curl "http://localhost:8088/query?key=q1" | jq '.result | has(0)' )
if [ "$result" == "true" ]; then
echo "has result"
else
echo "does not have result"
fi
See the manual. I wasn't able to get the -e or --exit-status arguments to work.

I'd use a language that can convert JSON to a native data structure:
wget -O- "http://localhost:8088/query?key=q1" |
perl -MJSON -0777 -ne '
$data = decode_json $_;
exit (#{$data->{result}} == 0)
'
That exits with a success status if the result attribute is NOT empty. Encapsulating into a shell function:
check_result() {
wget -O- "http://localhost:8088/query?key=q1" |
perl -MJSON -0777 -ne '$data = decode_json $_; exit (#{$data->{result}} == 0)'
}
if check_result; then
echo result is NOT empty
else
echo result is EMPTY
fi
I like ruby for parsing JSON:
ruby -rjson -e 'data = JSON.parse(STDIN.read); exit (data["result"].length > 0)'
It's interesting that the exit status requires the opposite comparison operator. I guess ruby will convert exit(true) to exit(0), unlike perl which has no true boolean objects only integers.

Related

How to update json field using jq [duplicate]

This question already has answers here:
Transfer or merge only some properties from one JSON file to another with jq
(2 answers)
jq to replace text directly on file (like sed -i)
(9 answers)
Closed 8 days ago.
I have a bootstrap script. After every time it runs, I want to keep track of its exit status in a JSON file. JSON file can have other fields too.
Case 1: The very first time it runs, JSON file will be created & a field node_bootstrap_status with boolean value will be added to it.
Case 2: In subsequent runs, JSON file will pre-exist. But in this case, I want to update same JSON field node_bootstrap_status with the outcome(again boolean).
I wrote the following bash script. It works for 1st case. In second case however, it ends up deleting everything in pre-existing JSON file.
exit_code=1
node_info_file="/etc/node_info.json"
if [[ -f $node_info_file ]]; then
# /etc/node_info.json exists
echo "${node_info_file} exists"
if [ $exit_code = 0 ]; then
cat $node_info_file | jq --argjson node_bootstrap_status true '{"node_bootstrap_status": $node_bootstrap_status}' > $node_info_file
else
cat $node_info_file | jq --argjson node_bootstrap_status false '{"node_bootstrap_status": $node_bootstrap_status}' > $node_info_file
fi
else
# /etc/node_info.json does NOT exists
echo "${node_info_file} does not exist"
touch ${node_info_file}
if [ $exit_code = 0 ]; then
echo '{}' | jq --argjson node_bootstrap_status true '{"node_bootstrap_status": $node_bootstrap_status}' > $node_info_file
else
echo '{}' | jq --argjson node_bootstrap_status false '{"node_bootstrap_status": $node_bootstrap_status}' > $node_info_file
fi
fi
expected outcome:
cat /etc/node_info.json
{
"node_bootstrap_status": true/false,
"foo": "bar"
}
You're off to a good start - but you'd want to use assignment rather than the sort of { ... } syntax like so:
node_info_file="node_info.json"
set_status() {
local value="$1"
mv -f "$node_info_file" "$node_info_file.tmp"
# Set property key with: '.node_bootstrap_status = ...'
jq --argjson node_bootstrap_status "$value" '.node_bootstrap_status = $node_bootstrap_status' "$node_info_file.tmp" > "$node_info_file"
rm -f "$node_info_file.tmp"
}
# if node_info.json does NOT exists; create it
if [ ! -f $node_info_file ]; then
printf '%s\n' '{}' > "$node_info_file"
fi
set_status true
set_status false
I tested this with empty JSON, and with other contents, and it worked as expected.
Also, make sure you quote "$node_info_file" - it's good practice. If you use ShellCheck then it'll catch those types of errors for you.
Try to adapt this version to your needs :
#!/usr/bin/env bash
exit_code=1
node_info_file="/etc/node_info.json"
test -f "$node_info_file" || echo {} > "$node_info_file"
[ $exit_code = 0 ] && status=true || status=false
tmp=$(mktemp)
jq --argjson node_bootstrap_status $status '{$node_bootstrap_status}' "$node_info_file" > $tmp
mv $tmp "$node_info_file"
Just a side note, it may not be a good idea to save node_info.json in /etc.

Check if result is an empty string

I have a JSON file which I created using a jq command.
The file is like this:
[
{
"description": "",
"app": "hello-test-app"
},
{
"description": "",
"app": "hello-world-app"
}
]
I would like to have just a simple if/else condition to check if the description is empty.
I tried different approaches but none of them works!!
I tried:
jq -c '.[]' input.json | while read i; do
description=$(echo $i | jq '.description')
if [[ "$description" == "" ]];
then
echo "$description is empty"
fi
done
and with same code but this if/else;
if [[ -z "$description" ]];
then
echo "$description is empty"
fi
Can someone help me?
jq supports conditionals. No need to bring this back to bash (yet):
< foo jq -r '.[] | if .description == ""
then "description is empty"
else .description end'
description is empty
description is empty
If you insist on piping back to bash, here's what is happening:
jq -c '.[]' foo | while read i; do description=$(echo $i | jq '.description')
printf '%s\n' "$description"
done
""
""
You can see here that the expansion of $description is not empty. It is literally a pair of quotes each time.
There are several problems with piping to bash here -- the unquoted expansion of $i, repiping to jq and translating a pair of quotes into a empty string between two different programming languages. But I guess the simple answer is "just test if "$description" expands to a pair of quotes."
Testing quotes in bash means quoting your quotes:
if [[ $description = '""' ]]; then
echo '$description expands to a pair of quotes'
fi
A better answer is, in my opinion, keep the work in jq.

How to use regex to parse this json: { "success" : true }?

I am using a shell script to make an API call and I need to verify that the json response is this:
{ "success" : true }
I am able to echo the call response to see that it has that value but I need to validate the response in an if statement so that the script can continue, I have tried to do this a number of ways with no success
Regex - I have used regex to extract values from other json responses, but I have not found a regex pattern that can extract the value of "success" with this json
String Comparison - I thought of simply using this condition to attempt to match the strings:
if [ "$callResponse" = '{ "success" : true }' ]
However I quickly ran into issues with the script reading the json due to its special characters, I tried using sed to add a backslash before each special character but sed could not read the json either
Lastly I tried to pipe the response to python but got the error "ValueError: No JSON object could be decoded" when using this command:
status=${$callResponse | python -c "import sys, json; print(json.load(sys.stdin)['success'])"}
Does anyone know a regex pattern that could find that specific json string? Is there another simple solution to this issue?
(Note that it is not possible to download jq or any other utilities for this PC)
Since the caller knows that the response is { "success" : true }, I can't think of any reason to not use jq in this case. For instance, you can try something like this:
if echo '{ "success" : true }' | jq --exit-status '.success == true' >/dev/null; then
echo "success"
# Do something success is true in the response.
else
echo "not success" response.
# Do something else success is not true or absent in the
fi
If you want to make an API call and get the response, you can easily pass the JSON response directly from wget to jq instead of going the roundabout way of storing it in an intermediate variable by tweaking it like this:
if wget --timeout 10 -O - -q -t 1 https://your.api.com/endpoint | jq --exit-status '.success == true' >/dev/null; then
echo "success"
else
echo "not success"
fi
To match when the value of success is true in a flexible way:
"success"\s*:\s*"?true"?
This will match all of these:
{ "success" : true }
{ "success" : "true" }
{ "success":true}
To be strict and match the above, but not imbalanced quotes like { "success" : "true }, use this:
"success"\s*:\s*("?)true\1
I would highly recommend not doing it that way.
We used to do it this way long ago and got into trouble when the response code is "200 OK" but receiving {"success": false} seemed to contradict each other.
A better approach is to use the response status codes instead.
Simply return 200 OK if success is true otherwise return the appropriate error status code if its not.
https://www.restapitutorial.com/httpstatuscodes.html
EDIT:
Bash script to help:
COOKIE_FILE="cookies.txt"
SERVER_IP="172.1.2.3"
LOGFILE="logs/api-calls.log"
WGETLOGFILE="logs/last-api-call.log"
#Helper function
on_wget_err ( )
{
EXITCODE=${1}
case ${EXITCODE} in
0) RESULT="OK";;
*) cat ${WGETLOGFILE} >> ${LOGFILE};
grep "HTTP/1.1" ${WGETLOGFILE} | gawk '{print substr($0,16)}'
exit 0;;
esac
}
if wget -O - -qT 4 -t 1 ${SERVER_IP} > /dev/null; then
echo "Server is up"
wget -S -O - --load-cookies ${COOKIE_FILE} "http://${SERVER_IP}${SERVER_ADDRESS}/my/api?param=$1" 2> ${WGETLOGFILE}
on_wget_err ${?}
echo "API was successfull"
else
echo "Server or network down"
exit 1;
fi
Your Python attempt was close. Here's a working one:
callResponse='{ "success" : true }'
status=$(echo "$callResponse" |
python -c "import sys, json; print(json.load(sys.stdin)['success'])")
echo "$status"
Or alternatively, rewritten to go straight in an if statement:
callResponse='{ "success" : true }'
if echo "$callResponse" | python -c "import sys, json; sys.exit(0 if json.load(sys.stdin)['success'] else 1)"
then
echo "Success"
fi

jq truncates ENV variable after whitespace

Trying to write a bash script that replaces values in a JSON file we are running into issues with Environment Variables that contain whitespaces.
Given an original JSON file.
{
"version": "base",
"myValue": "to be changed",
"channelId": 0
}
We want to run a command to update some variables in it, so that after we run:
CHANNEL_ID=1701 MY_VALUE="new value" ./test.sh
The JSON should look like this:
{
"version": "base",
"myValue": "new value",
"channelId": 1701
}
Our script is currently at something like this:
#!/bin/sh
echo $MY_VALUE
echo $CHANNEL_ID
function replaceValue {
if [ -z $2 ]; then echo "Skipping $1"; else jq --argjson newValue \"${2}\" '. | ."'${1}'" = $newValue' build/config.json > tmp.json && mv tmp.json build/config.json; fi
}
replaceValue channelId ${CHANNEL_ID}
replaceValue myValue ${MY_VALUE}
In the above all values are replaced by string and strings are getting truncated at whitespace. We keep alternating between this issue and a version of the code where substitutions just stop working entirely.
This is surely an issue with expansions but we would love to figure out, how we can:
- Replace values in the JSON with both strings and values.
- Use whitespaces in the strings we pass to our script.
You don't have to mess with --arg or --argjson to import the environment variables into jq's context. It can very well read the environment on its own. You don't need a script separately, just set the values along with the invocation of jq
CHANNEL_ID=1701 MY_VALUE="new value" \
jq '{"version": "base", myValue: env.MY_VALUE, channelId: env.CHANNEL_ID}' build/config.json
Note that in the case above, the variables need not be exported globally but just locally to the jq command. This allows you to not export multiple variables into the shell and pollute the environment, but just the ones needed for jq to construct the desired JSON.
To make the changes back to the original file, do > tmp.json && mv tmp.json build/config.json or more clearly download the sponge(1) utility from moreutils package. If present, you can pipe the output of jq as
| sponge build/config.json
Pass variables with --arg. Do:
jq --arg key "$1" --arg value "$2" '.[$key] = $value'
Notes:
#!/bin/sh indicates that this is posix shell script, not bash. Use #!/bin/bash in bash scripts.
function replaceValue { is something from ksh shell. Prefer replaceValue() { to declare functions. Bash obsolete and deprecated syntax.
Use newlines in your script to make it readable.
--argjson passes a json formatted argument, not a string. Use --arg for that.
\"${2}\" doesn't quote $2 expansion - it only appends and suffixes the string with ". Because the expansion is not qouted, word splitting is performed, which causes your input to be split on whitespaces when creating arguments for jq.
Remember to quote variable expansions.
Use http://shellcheck.net to check your scripts.
. | means nothing in jq, it's like echo $(echo $(echo))). You could jq '. | . | . | . | . | .' do it infinite number of times - it passes the same thing. Just write the thing you want to do.
Do:
#!/bin/bash
echo "$MY_VALUE"
echo "$CHANNEL_ID"
replaceValue() {
if [ -z "$2" ]; then
echo "Skipping $1"
else
jq --arg key "$1" --arg value "$2" '.[$key] = $value' build/config.json > tmp.json &&
mv tmp.json build/config.json
fi
}
replaceValue channelId "${CHANNEL_ID}"
replaceValue myValue "${MY_VALUE}"
#edit Replaced ."\($key)" with easier .[$key]
jq allows you to build new objects:
MY_VALUE=foo;
CHANNEL_ID=4
echo '{
"version": "base",
"myValue": "to be changed",
"channelId": 0
}' | jq ". | {\"version\": .version, \"myValue\": \"$MY_VALUE\", \"channelId\": $CHANNEL_ID}"
The . selects the whole input, and inputs that (|) to the construction of a new object (marked by {}). For version is selects .version from the input, but you can set your own values for the other two. We use double quotes to allow the Bash variable expansion, which means escaping the double quotes in the JSON.
You'll need to adapt my snippet above to scriptify it.

How can I interpret variables on the fly in the shell script?

I'm reading JSON in a shell script using JQ. Here, I'm unable to interpret the variables $HOME, $HOST, $PEMFILE in my shell script on the fly.
JSON File:
{
"script": {
"install": "${HOME}/lib/install.sh $HOST $PEMFILE",
"Setup": "${HOME}/lib/setup.sh $HOST $PEMFILE $VAR1 $VAR2"
}
}
Shell Script:
#!/bin/bash
examplefile="../lib/example.json"
HOST=ec2-..-...-...-...us-west-2.compute.amazonaws.com
PEMFILE=${HOME}/test.pem
installScript=($(jq '.script.install' $examplefile))
bash "$installScript"
Is there a way I can interpret these variables on the fly without modifying the JSON?
P.S I don't want to use eval.
It is easy using gnu utility envsubst:
installScript=$(jq -r '.script.install' "$examplefile" | envsubst)
Here is a solution using env and gsub to perform the replacement.
Note that env requires the variables to be passed as environment variables as opposed to shell variables.
#!/bin/bash
examplefile="../lib/example.json"
HOST=ec2-..-...-...-...us-west-2.compute.amazonaws.com
PEMFILE=${HOME}/test.pem
export HOST
export PEMFILE
installScript=$(jq -Mr '
.script.install | gsub("(?<x>[$][{]?\\w+[}]?)"; env[.x|gsub("[${}]+";"")] )
' $examplefile)
echo $installScript
Sample Output
/home/runner/lib/install.sh ec2-..-...-...-...us-west-2.compute.amazonaws.com /home/runner/test.pem
Try it online!
Specific solution
Here's a jq solution to the stated problem, though it will only work for "global" environment variables.
def substitute:
gsub("\\${HOME}"; env.HOME)
| gsub("\\$HOST"; env.HOST)
| gsub("\\$PEMFILE"; env.PEMFILE)
| gsub("\\$VAR1"; env.VAR1)
| gsub("\\$VAR2"; env.VAR2)
;
walk( if type=="string" then substitute else . end )
If your jq does not already have walk/1, then please either upgrade your jq or snarf the def from https://github.com/stedolan/jq/blob/master/src/builtin.jq
The solution above is a bit brittle but it could easily be robustified or generalized, as shown in the next section.
General solution
walk(if type == "string"
then gsub("\\$(?<x>[A-Za-z_][A-Za-z0-9_]+)"; "\(env[.x])")
| gsub("\\${(?<x>[A-Za-z_][A-Za-z0-9_]+)}"; "\(env[.x])")
else . end)
#!/bin/sh
TMP=$(mktemp /tmp/$$.XXX)
cat<<E_O_F > $TMP
cat <<EOF
$(cat so-dollar-variables.json)
EOF
E_O_F
. $TMP
/bin/rm "$TMP"
I've been hitting this on and off for years. I think I've finally got a decent pure-bash solution: uses regex matching and indirect parameter substitution
# read the file
json=$(< file.json)
echo step 0
echo "$json"
# set the relevant vars, just plain shell variables
HOST=_host_
PEMFILE=_pemfile_
VAR1=_var1_
VAR2=_var2_
# replace '$var' forms
while [[ $json =~ ("$"([[:alnum:]_]+)) ]]; do
json=${json//${BASH_REMATCH[1]}/${!BASH_REMATCH[2]}}
done;
echo
echo step 1
echo "$json"
# replace '${var}' forms
while [[ $json =~ ("$""{"([[:alnum:]_]+)"}") ]]; do
json=${json//${BASH_REMATCH[1]}/${!BASH_REMATCH[2]}}
done
echo
echo step 2
echo "$json"
Output
step 0
{
"script": {
"install": "${HOME}/lib/install.sh $HOST $PEMFILE",
"Setup": "${HOME}/lib/setup.sh $HOST $PEMFILE $VAR1 $VAR2"
}
}
step 1
{
"script": {
"install": "${HOME}/lib/install.sh _host_ _pemfile_",
"Setup": "${HOME}/lib/setup.sh _host_ _pemfile_ _var1_ _var2_"
}
}
step 2
{
"script": {
"install": "/home/jackman/lib/install.sh _host_ _pemfile_",
"Setup": "/home/jackman/lib/setup.sh _host_ _pemfile_ _var1_ _var2_"
}
}
The magic is:
the regular expression, where I capture both $VAR and VAR, and
[[ $json =~ ("$"([[:alnum:]_]+)) ]]
# ..........1 2 21
the parameter substitution, where I search for the string "$VAR" and replace it with the indirect variable expansion ${!VAR}
${json//${BASH_REMATCH[1]}/${!BASH_REMATCH[2]}}