How can I interpret variables on the fly in the shell script? - json

Question

I'm reading JSON in a shell script using JQ. Here, I'm unable to interpret the variables $HOME, $HOST, $PEMFILE in my shell script on the fly.
JSON File:
{
"script": {
"install": "${HOME}/lib/install.sh $HOST $PEMFILE",
"Setup": "${HOME}/lib/setup.sh $HOST $PEMFILE $VAR1 $VAR2"
}
}
Shell Script:
#!/bin/bash
examplefile="../lib/example.json"
HOST=ec2-..-...-...-...us-west-2.compute.amazonaws.com
PEMFILE=${HOME}/test.pem
installScript=($(jq '.script.install' $examplefile))
bash "$installScript"
Is there a way I can interpret these variables on the fly without modifying the JSON?
P.S I don't want to use eval.

Answer 1

It is easy using gnu utility envsubst:
installScript=$(jq -r '.script.install' "$examplefile" | envsubst)

Answer 2

Here is a solution using env and gsub to perform the replacement.
Note that env requires the variables to be passed as environment variables as opposed to shell variables.
#!/bin/bash
examplefile="../lib/example.json"
HOST=ec2-..-...-...-...us-west-2.compute.amazonaws.com
PEMFILE=${HOME}/test.pem
export HOST
export PEMFILE
installScript=$(jq -Mr '
.script.install | gsub("(?<x>[$][{]?\\w+[}]?)"; env[.x|gsub("[${}]+";"")] )
' $examplefile)
echo $installScript
Sample Output
/home/runner/lib/install.sh ec2-..-...-...-...us-west-2.compute.amazonaws.com /home/runner/test.pem
Try it online!

Answer 3

Specific solution
Here's a jq solution to the stated problem, though it will only work for "global" environment variables.
def substitute:
gsub("\\${HOME}"; env.HOME)
| gsub("\\$HOST"; env.HOST)
| gsub("\\$PEMFILE"; env.PEMFILE)
| gsub("\\$VAR1"; env.VAR1)
| gsub("\\$VAR2"; env.VAR2)
;
walk( if type=="string" then substitute else . end )
If your jq does not already have walk/1, then please either upgrade your jq or snarf the def from https://github.com/stedolan/jq/blob/master/src/builtin.jq
The solution above is a bit brittle but it could easily be robustified or generalized, as shown in the next section.
General solution
walk(if type == "string"
then gsub("\\$(?<x>[A-Za-z_][A-Za-z0-9_]+)"; "\(env[.x])")
| gsub("\\${(?<x>[A-Za-z_][A-Za-z0-9_]+)}"; "\(env[.x])")
else . end)

Answer 4

#!/bin/sh
TMP=$(mktemp /tmp/$$.XXX)
cat<<E_O_F > $TMP
cat <<EOF
$(cat so-dollar-variables.json)
EOF
E_O_F
. $TMP
/bin/rm "$TMP"

Answer 5

I've been hitting this on and off for years. I think I've finally got a decent pure-bash solution: uses regex matching and indirect parameter substitution
# read the file
json=$(< file.json)
echo step 0
echo "$json"
# set the relevant vars, just plain shell variables
HOST=_host_
PEMFILE=_pemfile_
VAR1=_var1_
VAR2=_var2_
# replace '$var' forms
while [[ $json =~ ("$"([[:alnum:]_]+)) ]]; do
json=${json//${BASH_REMATCH[1]}/${!BASH_REMATCH[2]}}
done;
echo
echo step 1
echo "$json"
# replace '${var}' forms
while [[ $json =~ ("$""{"([[:alnum:]_]+)"}") ]]; do
json=${json//${BASH_REMATCH[1]}/${!BASH_REMATCH[2]}}
done
echo
echo step 2
echo "$json"
Output
step 0
{
"script": {
"install": "${HOME}/lib/install.sh $HOST $PEMFILE",
"Setup": "${HOME}/lib/setup.sh $HOST $PEMFILE $VAR1 $VAR2"
}
}
step 1
{
"script": {
"install": "${HOME}/lib/install.sh _host_ _pemfile_",
"Setup": "${HOME}/lib/setup.sh _host_ _pemfile_ _var1_ _var2_"
}
}
step 2
{
"script": {
"install": "/home/jackman/lib/install.sh _host_ _pemfile_",
"Setup": "/home/jackman/lib/setup.sh _host_ _pemfile_ _var1_ _var2_"
}
}
The magic is:
the regular expression, where I capture both $VAR and VAR, and
[[ $json =~ ("$"([[:alnum:]_]+)) ]]
# ..........1 2 21
the parameter substitution, where I search for the string "$VAR" and replace it with the indirect variable expansion ${!VAR}
${json//${BASH_REMATCH[1]}/${!BASH_REMATCH[2]}}