Im checking inside a openwrt with very few shell commands to see If is possible to filter json string to have the values.
For example
{"address":"192.168.2.2","user":"user1","groups":"permissions"}
I receive from curl the string and I need to separate values to pass vars to other commands.
For now Im checking some examples but not works
#!/bin/sh
. /usr/share/libubox/jshn.sh
json_init
json_load '$(cat $STRING)'
json_get_keys keys
for k in $keys; do
json_get_var v "$k"
echo "$k : $v"
done
But produce error "Failed to parse message data"
My problem is justly that I cna't use jq, or python to choose data, so only solution is to separate first.
Suggestions?
I found other form more clean to do the same
eval $(jsonfilter -s $STRING -e 'ADDRESS=#.address' -e 'USER=#.user')
echo "address=$ADDRESS user=$USER"
With this form I can filter every value how parameter, without jq or python function.
Related
I'm doing the following to capture some ADO JSON data:
iteration="$(az boards iteration team list --team Test --project Test --timeframe current)"
Normally, the output of that command contains a JSON key/value pair like the following:
"path": "Test\\Sprint1"
But after capturing the STDOUT into that iteration variable, if I do
echo "$iteration"
That key/value pair becomes
"path": "Test\Sprint1"
And if I attempt to use jq on that output, it breaks because it's not recognized as valid JSON any longer. I'm very unfamiliar with Bash. How can I get that JSON to remain valid all the way through?
As already commented by markp-fuso:
It looks like your echo command is interpreting the backslashes. You can confirm this by running echo 'a\\b' and looking at the output.
The portable way to deal with such problems is to use printf instead of echo:
printf %s\\n "$iteration"
I'm creating a gitlab-ci.yml file where I need to iterate in a json file and extract the keys and values. I cannot use jq and I am trying with a cat command to do this. This is my script:
script:
while read line
do
echo $line
done < $myfile
And this is myfile:
{"var1":"test1",
"other_var":"test2"}
Since I am not able to use jq becuase is not installed and my customer doesn't allow me to install it, how I can make a print like this one:
"Key is var1 and value is test1"
I working on a function which extract the choosen track from a media container (mkv,mp4...etc). One of its major feature will be the "auto output file extension assigner".
the process will be the following...
step 1) when i give the script the number of the track, which i want to extract, it automatically inspect the source file with mediainfo and output the results in JSON format.
step 2) With JQ, i query the value of the "track" key from the selected track, and save it to the "mediaFormat" variable.
step 3) put this variable in a switch statement and compare with a predefined list of switches. If there is a match, then it will initialize the "mediaExtension" variable
with the appropriate value, which will be used as a extension of the ouput file.
For now i just want echo the "mediaExtension" variable, to see if it works. And it DIDN'T WORK.
The problem is step 1-2 works as expected, but somehow the switch statement (step 3) doesn't work. Only the (*) switch will be executed, which means it doesn't recognize the "AVC" switch.
#!/bin/bash
# INCLUDES
# mediainfo binary
PATH=/cygdrive/c/build_suite/local64/bin-video:$PATH;
# jq binary
PATH=/cygdrive/c/build_suite/local64/bin-global:$PATH;
# BASH SETTINGS
set -x;
# FUNCTION PARAMETER
function inspectExtension () {
mediaFormat=$(mediainfo "$1" --Output=JSON | jq ".media.track[$2].Format");
case $mediaFormat in
"AVC") mediaExtension="264";;
*) echo "ERROR";;
esac
set "$mediaExtension";
echo "$mediaExtension";
}
inspectExtension "test.mp4" "1";
read -p "Press enter to continue...";
And as you can see, in this script i activated tracing (set -x), and this is what i see in the console window (i use cygwin on windows 10).
+ inspectExtension test.mp4 1
++ mediainfo test.mp4 --Output=JSON
++ jq '.media.track[1].Format'
' mediaFormat='"AVC"
+ case $mediaFormat in
+ echo ERROR
ERROR
+ set ''
+ echo ''
+ read -p 'Press enter to continue...'
Press enter to continue...
Any ideas? Or is something what i miss here?
Thx for the help!
Maybe the only thing you miss is using the --raw-output option of jq like so:
mediaFormat=$(mediainfo "$1" --Output=JSON | jq --raw-output ".media.track[$2].Format");
Whenever you use jq to access some string values, it will be best to use the --raw-output option because it get's rid of the enclosing quotes.
Assuming you want mediaFormat to be a JSON value (i.e., assuming the invocation of jq is the way you have it), "AVC" in the case statement should be quoted:
'"AVC"' ) ...
In addition, it would probably be safer to quote the argument of case.
I wrote a script to create sqs resource on local stack. I wanted to pass a value that I get from one cli command to the next but inside an inline json. Following is the section of the script in question.
arn=$(aws --endpoint-url=http://localhost:4576 sqs get-queue-attributes \
--queue-url http://localhost:4576/my_dead_letter_queue_url \
--query 'Attributes.QueueArn' \
--output text)
aws --endpoint-url=http://localhost:4576 sqs create-queue \
--queue-name my_queue \
--attributes \
'{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
So I'm trying to pass that "arn" variable but the cli is taking that as a string and trying to find a sqs with url "$arn" and fails. I also tried removing the quote. In that case, the error is malformed string.
Instead of the arn variable, if I use the arn value as string there, it works.
Can someone please show me how to pass that variable inside that inline json if it is possible?
Thank you for reading :)
Shahed
I was able to do the following with successful results, grant it it doesn't process the json (for that I'm just replacing tokens via sed), but I updated my example and tested it at least in bash with what I was doing:
#!/bin/bash
export awscmd="aws --region us-east-1 iam"
function setArn() {
${awscmd} list-policies --query 'Policies[?PolicyName==`'${1}'`].{ARN:Arn}' --output text
}
arn=$(setArn "some-policy-name")
echo '{"RedrivePolicy":"{"deadLetterTargetArn":"'$arn'", "maxReceiveCount":"5"}"}'
$ ./somearntest.sh
{"RedrivePolicy":"{"deadLetterTargetArn":"arn:aws:iam::############:policy/some-policy-name", "maxReceiveCount":"5"}"}
Notice the use of single tics to concatenate the output result outside of the string. This is in bash 4 and I removed the escaped \"s as I think that was added in error; ymmv.
The problem here is you are trying to expand a bash variable inside single quotes. Using single quotes like this is usually to pass a bunch of strings and unqoutable stuff as one argument. If you can't replace them with double quotes you'll have to resort to dirty eval hacks, which I do not recommend.
Here is an example:
$ arn=foobar
$ echo '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}
$ eval echo '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
{RedrivePolicy:{"deadLetterTargetArn":"foobar", "maxReceiveCount":"5"}}
For more information I suggest to check How eval works and Expansion of variables inside single quotes
I need to catch an error when lifting a service. The response can be null, a string error message like
error services-migration/foobar: Not found: services-migration/foobar
or a valid JSON when everything is fine. I was wondering if there is a way with jq to simply check if the provided string is a valid JSON. I could ofc check the string for some keywords like error f.e., but I'm looking for a more robust option, where eg. I get a true/false or 1/0 from jq.
I was looking through the docs of jq and also some questions here on SO but everything was about finding and picking out key-values from a JSON, but nothing about simply validating a string.
UPDATE:
I've got this:
result=$(some command)
from which the result is the string error services-migration/foobar: Not found: services-migration/foobar
And then the if statement:
if jq -e . >/dev/null 2>&1 <<<"$result"; then
echo "it catches it"
else
echo "it doesn't catch it"
fi
And it always ends up in the else clause.
From the manual:
-e / --exit-status:
Sets the exit status of jq to 0 if the last output values was neither false nor null, 1 if the last output value was either false or null, or 4 if no valid result was ever produced. Normally jq exits with 2 if there was any usage problem or system error, 3 if there was a jq program compile error, or 0 if the jq program ran.
So you can use:
if jq -e . >/dev/null 2>&1 <<<"$json_string"; then
echo "Parsed JSON successfully and got something other than false/null"
else
echo "Failed to parse JSON, or got false/null"
fi
In fact, if you don't care about distinguishing between the different types of error, then you can just lose the -e switch. In this case, anything considered to be valid JSON (including false/null) will be parsed successfully by the filter . and the program will terminate successfully, so the if branch will be followed.
This is working for me
echo $json_string | jq -e . >/dev/null 2>&1 | echo ${PIPESTATUS[1]}
that returns return code:
0 - Success
1 - Failed
4 - Invalid
Then you can evaluate the return code by further code.
The commands and explanations below attempt to use jq in the least expensive way, just as means to validate JSON. In my opinion, there are tools better suited for this (Like JSONLint), but if the only thing you have at hand is jq then this is my contribution.
To validate a JSON file and still get the errors in STDERR output using jq you can use the following bash one-liner
jq -reM '""' <<<'<MAYBE_INVALID_JSON>' 1>/dev/null
Which reads as:
Execute jq with flags:
-r to output a raw value (No processing or formatting)
-e to exit with a code greater than 0 if there was an error
-M do not colorize JSON
The first jq argument is '""' which outputs an empty string, effectively preventing jq from making any processing of the JSON for printing
<<< followed by a string is called a "Here String", and is basically telling bash to treat the string as if it was a file and pass it to STDIN (As input to the command).
Note that you can replace <<<'<MAYBE_INVALID_JSON>' with the path of a file you want to validate and it would still work
Then redirect any STDOUT output (Which normally will contain the JSON) to /dev/null, effectively discarding it
You can also get fancier and output a custom message below the errors by using the following:
jq -reM '""' <<<'<MAYBE_INVALID_JSON>' 1>/dev/null || ( exit_code="$?"; echo 'ERROR: Invalid JSON file. See errors above' 1>&2; exit "$exit_code" )
Where the added portion reads as follows:
|| means: If the previous command failed with a non-zero exit code then run the following command
Open a sub-shell with (. This is actually needed since we want to exit with the original command's exit code
Set the exit_code environment variable to the exit code of the last command $?
Print custom error message using echo
exit the sub-shell with the exit code from the original command $exit_code
Close the sub-shell
For a variable content like for a line, that's been working for me
echo "$my_potential_json"|jq empty > /dev/null 2>&1
if [ $? -eq 0 ]; then
echo "found valid json:$my_potential_json"
do_whatever....