I have this code
#!/bin/bash
today=$(date +'%Y-%m-%d')
date_in_config=$(cat config.json | jq ".date")
echo $date_in_config
echo $today
if [ $date_in_config == $today ];then
echo "Same date"
else
echo "different"
fi
config.json
{
"date": "2021-03-24",
"session": "0"
}
I am stuck at comparing the dates. The output that I get from the above code is
"2021-03-24"
2021-03-24
different
What am I doing wrong?
Clearly you have already detected what is wrong: For Bash (and any other programming language I can think of), "x" is different from x.
Put the double-quotes in today:
today=$(date +'"%Y-%m-%d"')
Or remove the quotes when retrieving the JSON information:
date_in_config=$(jq -r ".date" < config.json)
And remember to quote your variable expansions, as Shellcheck would have told you. Also get rid of the == bashism (optional, for portability).
[ "$date_in_config" = "$today" ]
Related
I have a JSON file which I created using a jq command.
The file is like this:
[
{
"description": "",
"app": "hello-test-app"
},
{
"description": "",
"app": "hello-world-app"
}
]
I would like to have just a simple if/else condition to check if the description is empty.
I tried different approaches but none of them works!!
I tried:
jq -c '.[]' input.json | while read i; do
description=$(echo $i | jq '.description')
if [[ "$description" == "" ]];
then
echo "$description is empty"
fi
done
and with same code but this if/else;
if [[ -z "$description" ]];
then
echo "$description is empty"
fi
Can someone help me?
jq supports conditionals. No need to bring this back to bash (yet):
< foo jq -r '.[] | if .description == ""
then "description is empty"
else .description end'
description is empty
description is empty
If you insist on piping back to bash, here's what is happening:
jq -c '.[]' foo | while read i; do description=$(echo $i | jq '.description')
printf '%s\n' "$description"
done
""
""
You can see here that the expansion of $description is not empty. It is literally a pair of quotes each time.
There are several problems with piping to bash here -- the unquoted expansion of $i, repiping to jq and translating a pair of quotes into a empty string between two different programming languages. But I guess the simple answer is "just test if "$description" expands to a pair of quotes."
Testing quotes in bash means quoting your quotes:
if [[ $description = '""' ]]; then
echo '$description expands to a pair of quotes'
fi
A better answer is, in my opinion, keep the work in jq.
Trying to write a bash script that replaces values in a JSON file we are running into issues with Environment Variables that contain whitespaces.
Given an original JSON file.
{
"version": "base",
"myValue": "to be changed",
"channelId": 0
}
We want to run a command to update some variables in it, so that after we run:
CHANNEL_ID=1701 MY_VALUE="new value" ./test.sh
The JSON should look like this:
{
"version": "base",
"myValue": "new value",
"channelId": 1701
}
Our script is currently at something like this:
#!/bin/sh
echo $MY_VALUE
echo $CHANNEL_ID
function replaceValue {
if [ -z $2 ]; then echo "Skipping $1"; else jq --argjson newValue \"${2}\" '. | ."'${1}'" = $newValue' build/config.json > tmp.json && mv tmp.json build/config.json; fi
}
replaceValue channelId ${CHANNEL_ID}
replaceValue myValue ${MY_VALUE}
In the above all values are replaced by string and strings are getting truncated at whitespace. We keep alternating between this issue and a version of the code where substitutions just stop working entirely.
This is surely an issue with expansions but we would love to figure out, how we can:
- Replace values in the JSON with both strings and values.
- Use whitespaces in the strings we pass to our script.
You don't have to mess with --arg or --argjson to import the environment variables into jq's context. It can very well read the environment on its own. You don't need a script separately, just set the values along with the invocation of jq
CHANNEL_ID=1701 MY_VALUE="new value" \
jq '{"version": "base", myValue: env.MY_VALUE, channelId: env.CHANNEL_ID}' build/config.json
Note that in the case above, the variables need not be exported globally but just locally to the jq command. This allows you to not export multiple variables into the shell and pollute the environment, but just the ones needed for jq to construct the desired JSON.
To make the changes back to the original file, do > tmp.json && mv tmp.json build/config.json or more clearly download the sponge(1) utility from moreutils package. If present, you can pipe the output of jq as
| sponge build/config.json
Pass variables with --arg. Do:
jq --arg key "$1" --arg value "$2" '.[$key] = $value'
Notes:
#!/bin/sh indicates that this is posix shell script, not bash. Use #!/bin/bash in bash scripts.
function replaceValue { is something from ksh shell. Prefer replaceValue() { to declare functions. Bash obsolete and deprecated syntax.
Use newlines in your script to make it readable.
--argjson passes a json formatted argument, not a string. Use --arg for that.
\"${2}\" doesn't quote $2 expansion - it only appends and suffixes the string with ". Because the expansion is not qouted, word splitting is performed, which causes your input to be split on whitespaces when creating arguments for jq.
Remember to quote variable expansions.
Use http://shellcheck.net to check your scripts.
. | means nothing in jq, it's like echo $(echo $(echo))). You could jq '. | . | . | . | . | .' do it infinite number of times - it passes the same thing. Just write the thing you want to do.
Do:
#!/bin/bash
echo "$MY_VALUE"
echo "$CHANNEL_ID"
replaceValue() {
if [ -z "$2" ]; then
echo "Skipping $1"
else
jq --arg key "$1" --arg value "$2" '.[$key] = $value' build/config.json > tmp.json &&
mv tmp.json build/config.json
fi
}
replaceValue channelId "${CHANNEL_ID}"
replaceValue myValue "${MY_VALUE}"
#edit Replaced ."\($key)" with easier .[$key]
jq allows you to build new objects:
MY_VALUE=foo;
CHANNEL_ID=4
echo '{
"version": "base",
"myValue": "to be changed",
"channelId": 0
}' | jq ". | {\"version\": .version, \"myValue\": \"$MY_VALUE\", \"channelId\": $CHANNEL_ID}"
The . selects the whole input, and inputs that (|) to the construction of a new object (marked by {}). For version is selects .version from the input, but you can set your own values for the other two. We use double quotes to allow the Bash variable expansion, which means escaping the double quotes in the JSON.
You'll need to adapt my snippet above to scriptify it.
I have a json in which I want to modify a particular value but the terminal always displays the json with the modified value but it does not actually change the value in the particular file. Sample json:
{
name: 'abcd',
age: 30,
address: 'abc'
}
I want to change the value of address in the file itself but so far I've been unable to do so. I tried using:
jq '.address = "abcde"' test.json
but it didn't work. Any suggestions?
Use a temporary file; it's what any program that claims to do in-place editing is doing.
tmp=$(mktemp)
jq '.address = "abcde"' test.json > "$tmp" && mv "$tmp" test.json
If the address isn't hard-coded, pass the correct address via a jq argument:
address=abcde
jq --arg a "$address" '.address = $a' test.json > "$tmp" && mv "$tmp" test.json
AFAIK jq does not support in-place editing, so you must redirect to a temporary file first and then replace your original file with it, or use sponge utility from the moreutils package, like that:
jq '.address = "abcde"' test.json|sponge test.json
There are other techniques to "redirect to the same file", like saving your output in a variable e.t.c. "Unix & Linux StackExchange" is a good place to start, if you want to learn more about this.
Temp files add more complexity when not needed (unless you are truly dealing with JSON files so large you cannot fit them in memory (GB to 100's of GB or TB, depending on how much RAM/parallelism you have)
The Pure bash way.
contents="$(jq '.address = "abcde"' test.json)" && \
echo -E "${contents}" > test.json
Pros
No temp file to juggle
Pure bash
Don't need an admin to install sponge, which is not installed by default
Simpler
Cons
This works perfectly fine for json because it cannot contain a literal null character. If you were to try this outside the json arena, it would fail when a null is encountered (and you would have to do some encoding/decoding workarounds). Bash variables cannot store literal nulls.
Note: this can not be combined as "one command" (like #codekandis
suggested), since redirection sometimes starts before the left hand side (LHS) of an expression is run, and starting redirection before running jq erroneously empties the file, hence two separate commands. It may "seem" to work when you try it, but this is misleading and has a very high probability of failing as soon as the circumstances change.
Update: Added -E option to disable escape characters just in case you are on systems where they are interpreted by default. (Which I've never actually seen)
Just to add to chepner answer and if you want it in a shell script.
test.json
{
"name": "abcd",
"age": 30,
"address": "abc"
}
script.sh
#!/bin/bash
address="abcde"
age=40
# Strings:
jq --arg a "${address}" '.address = $a' test.json > "tmp" && mv "tmp" test.json
# Integers:
jq --argjson a "${age}" '.age = $a' test.json > "tmp" && mv "tmp" test.json
Example for nested json with changing single and multiple values.
config.json
{
"Parameters": {
"Environment": "Prod",
"InstanceType": "t2.micro",
"AMIID": "ami-02d8e11",
"ConfigRegion": "eu-west-1"
}
}
with the below command, you can edit multiple values.
tmp=$(mktemp)
jq '.Parameters.AMIID = "ami-02d8sdfsdf" | .Parameters.Environment = "QA"' config.json > "$tmp" && mv "$tmp" config.json
with the below command, you can edit single value.
tmp=$(mktemp)
jq '.Parameters.AMIID = "ami-02d8sdfsdf"' config.json > "$tmp" && mv "$tmp" config.json
this should work
address = aaaaa
echo $(jq --arg a "$address" '.address = ($a)' test.json) > test.json
for whatever reason, without the echo, it makes a bin file and my python script was not able to parse it.
I took the best of a couple answers here and here.
This uses a parameter named actionname as an input to an assignment of the name property at the document level. ACTION_NAME is just an envvar I want to use as the replacement value.
contents="$(jq --arg actionname ${ACTION_NAME} '.name = $actionname' ./${ACTION_NAME}/package.json)" && \
echo -E "${contents}" > ${ACTION_NAME}/package.json;
I didn't like any of the solutions and created the sde utility.
pip install sde
Then, e.g. for the following JSON data:
{
"Parameters": {
"Environment": "Prod",
"InstanceType": "t2.micro",
"AMIID": "ami-02d8e11",
"ConfigRegion": "eu-west-1"
}
}
you can simply do:
sde Parameters.Environment Dev test.json
I have a json file names test.json with the below content.
{
"run_list": ["recipe[cookbook-ics-op::setup_server]"],
"props": {
"install_home": "/test/inst1",
"tmp_dir": "/test/inst1/tmp",
"user": "tuser
}
}
I want to read this file into a variable in shell script & then extract the values of install_home,user & tmp_dir using expr. Can someone help, please?
props=cat test.json
works to get the json file into a variable. Now how can I extract the values using expr. Any help would be greatly appreciated.
Install jq
yum -y install epel-release
yum -y install jq
Get the values in the following way
install_home=$(cat test.json | jq -r '.props.install_home')
tmp_dir=$(cat test.json | jq -r '.props.tmp_dir')
user=$(cat test.json | jq -r '.props.user')
For a pure bash solution I suggest this:
github.com/dominictarr/JSON.sh
It could be used like this:
./json.sh -l -p < example.json
print output like:
["name"] "JSON.sh"
["version"] "0.2.1"
["description"] "JSON parser written in bash"
["homepage"] "http://github.com/dominictarr/JSON.sh"
["repository","type"] "git"
["repository","url"] "https://github.com/dominictarr/JSON.sh.git"
["bin","JSON.sh"] "./JSON.sh"
["author"] "Dominic Tarr <dominic.tarr#gmail.com> (http://bit.ly/dominictarr)"
["scripts","test"] "./all-tests.sh"
From here is pretty trivial achive what you are looking for
jq is a dedicated parser for JSON files. Install jq.
values in the json can be retrieved as:
jq .<top-level-attr>.<next-level-attr> <json-file-path>
if JSON contains an array
jq .<top-level-attr>.<next-level-array>[].<elem-in-array> <json-file-path>
if you want a value in a shell variable
id = $(jq -r .<top-level-attr>.<next-level-array>[].<next-level-attr> <json-file-path>)
echo id
use -r if you need unquoted value
For simple JSON, it may be treated as a plain text file.
In that case, we can use simple text pattern matching to extract the information we need.
If you observe the following lines:
"install_home": "/test/inst1",
"tmp_dir": "/test/inst1/tmp",
"user": "user"
There exists a pattern on each line that can be described as key and value:
"key" : "value"
We can use perl with regular expressions to exact the value for any given key:
"key" hardcoded for each case "install_home", "tmp_dir" and "user"
"value" as (.*) regular expression
Then we use the $1 matching group to retrieve the value.
i=$(perl -ne 'if (/"install_home": "(.*)"/) { print $1 . "\n" }' test.json)
t=$(perl -ne 'if (/"tmp_dir": "(.*)"/) { print $1 . "\n" }' test.json)
u=$(perl -ne 'if (/"user": "(.*)"/) { print $1 . "\n" }' test.json)
cat <<EOF
install_home: $i
tmp_dir : $t
user : $u
EOF
Which outputs:
install_home: /test/inst1
tmp_dir : /test/inst1/tmp
user : tuser
So, lets say, I am trying to write a shell script which does the following:
1) Ping http://localhost:8088/query?key=q1
2) It returns a json response :
{
"status": "success",
"result": []
"query": "q1"
}
or
{
"status": "success",
"result": ["foo"],
"artist_key": "q1"
}
The "result" is either an empty array or filled array..
I am trying to write a shell script which is checking whether "result" is empty list or not?
Something like this would work:
# Assume result is from curl, but could be in a file or whatever
if curl "http://localhost:8088/query?key=q1" | grep -Pq '"result":\s+\[".+"\]'; then
echo "has result"
else
echo "does not have result"
fi
However, I'm assuming these are on separate lines. If not, there are linters for format it.
Edited (based on the jq comment), here's a jq solution as suggested by Adrian Frühwirth:
result=$( curl "http://localhost:8088/query?key=q1" | jq '.result' )
if [ "$result" == "[]" ]; then
echo "does not have result"
else
echo "has result"
fi
I have learned something new today. And as I play around with this more, maybe it's better to do this:
result=$( curl "http://localhost:8088/query?key=q1" | jq '.result | has(0)' )
if [ "$result" == "true" ]; then
echo "has result"
else
echo "does not have result"
fi
See the manual. I wasn't able to get the -e or --exit-status arguments to work.
I'd use a language that can convert JSON to a native data structure:
wget -O- "http://localhost:8088/query?key=q1" |
perl -MJSON -0777 -ne '
$data = decode_json $_;
exit (#{$data->{result}} == 0)
'
That exits with a success status if the result attribute is NOT empty. Encapsulating into a shell function:
check_result() {
wget -O- "http://localhost:8088/query?key=q1" |
perl -MJSON -0777 -ne '$data = decode_json $_; exit (#{$data->{result}} == 0)'
}
if check_result; then
echo result is NOT empty
else
echo result is EMPTY
fi
I like ruby for parsing JSON:
ruby -rjson -e 'data = JSON.parse(STDIN.read); exit (data["result"].length > 0)'
It's interesting that the exit status requires the opposite comparison operator. I guess ruby will convert exit(true) to exit(0), unlike perl which has no true boolean objects only integers.