I working on a function which extract the choosen track from a media container (mkv,mp4...etc). One of its major feature will be the "auto output file extension assigner".
the process will be the following...
step 1) when i give the script the number of the track, which i want to extract, it automatically inspect the source file with mediainfo and output the results in JSON format.
step 2) With JQ, i query the value of the "track" key from the selected track, and save it to the "mediaFormat" variable.
step 3) put this variable in a switch statement and compare with a predefined list of switches. If there is a match, then it will initialize the "mediaExtension" variable
with the appropriate value, which will be used as a extension of the ouput file.
For now i just want echo the "mediaExtension" variable, to see if it works. And it DIDN'T WORK.
The problem is step 1-2 works as expected, but somehow the switch statement (step 3) doesn't work. Only the (*) switch will be executed, which means it doesn't recognize the "AVC" switch.
#!/bin/bash
# INCLUDES
# mediainfo binary
PATH=/cygdrive/c/build_suite/local64/bin-video:$PATH;
# jq binary
PATH=/cygdrive/c/build_suite/local64/bin-global:$PATH;
# BASH SETTINGS
set -x;
# FUNCTION PARAMETER
function inspectExtension () {
mediaFormat=$(mediainfo "$1" --Output=JSON | jq ".media.track[$2].Format");
case $mediaFormat in
"AVC") mediaExtension="264";;
*) echo "ERROR";;
esac
set "$mediaExtension";
echo "$mediaExtension";
}
inspectExtension "test.mp4" "1";
read -p "Press enter to continue...";
And as you can see, in this script i activated tracing (set -x), and this is what i see in the console window (i use cygwin on windows 10).
+ inspectExtension test.mp4 1
++ mediainfo test.mp4 --Output=JSON
++ jq '.media.track[1].Format'
' mediaFormat='"AVC"
+ case $mediaFormat in
+ echo ERROR
ERROR
+ set ''
+ echo ''
+ read -p 'Press enter to continue...'
Press enter to continue...
Any ideas? Or is something what i miss here?
Thx for the help!
Maybe the only thing you miss is using the --raw-output option of jq like so:
mediaFormat=$(mediainfo "$1" --Output=JSON | jq --raw-output ".media.track[$2].Format");
Whenever you use jq to access some string values, it will be best to use the --raw-output option because it get's rid of the enclosing quotes.
Assuming you want mediaFormat to be a JSON value (i.e., assuming the invocation of jq is the way you have it), "AVC" in the case statement should be quoted:
'"AVC"' ) ...
In addition, it would probably be safer to quote the argument of case.
Related
Im checking inside a openwrt with very few shell commands to see If is possible to filter json string to have the values.
For example
{"address":"192.168.2.2","user":"user1","groups":"permissions"}
I receive from curl the string and I need to separate values to pass vars to other commands.
For now Im checking some examples but not works
#!/bin/sh
. /usr/share/libubox/jshn.sh
json_init
json_load '$(cat $STRING)'
json_get_keys keys
for k in $keys; do
json_get_var v "$k"
echo "$k : $v"
done
But produce error "Failed to parse message data"
My problem is justly that I cna't use jq, or python to choose data, so only solution is to separate first.
Suggestions?
I found other form more clean to do the same
eval $(jsonfilter -s $STRING -e 'ADDRESS=#.address' -e 'USER=#.user')
echo "address=$ADDRESS user=$USER"
With this form I can filter every value how parameter, without jq or python function.
I need to catch an error when lifting a service. The response can be null, a string error message like
error services-migration/foobar: Not found: services-migration/foobar
or a valid JSON when everything is fine. I was wondering if there is a way with jq to simply check if the provided string is a valid JSON. I could ofc check the string for some keywords like error f.e., but I'm looking for a more robust option, where eg. I get a true/false or 1/0 from jq.
I was looking through the docs of jq and also some questions here on SO but everything was about finding and picking out key-values from a JSON, but nothing about simply validating a string.
UPDATE:
I've got this:
result=$(some command)
from which the result is the string error services-migration/foobar: Not found: services-migration/foobar
And then the if statement:
if jq -e . >/dev/null 2>&1 <<<"$result"; then
echo "it catches it"
else
echo "it doesn't catch it"
fi
And it always ends up in the else clause.
From the manual:
-e / --exit-status:
Sets the exit status of jq to 0 if the last output values was neither false nor null, 1 if the last output value was either false or null, or 4 if no valid result was ever produced. Normally jq exits with 2 if there was any usage problem or system error, 3 if there was a jq program compile error, or 0 if the jq program ran.
So you can use:
if jq -e . >/dev/null 2>&1 <<<"$json_string"; then
echo "Parsed JSON successfully and got something other than false/null"
else
echo "Failed to parse JSON, or got false/null"
fi
In fact, if you don't care about distinguishing between the different types of error, then you can just lose the -e switch. In this case, anything considered to be valid JSON (including false/null) will be parsed successfully by the filter . and the program will terminate successfully, so the if branch will be followed.
This is working for me
echo $json_string | jq -e . >/dev/null 2>&1 | echo ${PIPESTATUS[1]}
that returns return code:
0 - Success
1 - Failed
4 - Invalid
Then you can evaluate the return code by further code.
The commands and explanations below attempt to use jq in the least expensive way, just as means to validate JSON. In my opinion, there are tools better suited for this (Like JSONLint), but if the only thing you have at hand is jq then this is my contribution.
To validate a JSON file and still get the errors in STDERR output using jq you can use the following bash one-liner
jq -reM '""' <<<'<MAYBE_INVALID_JSON>' 1>/dev/null
Which reads as:
Execute jq with flags:
-r to output a raw value (No processing or formatting)
-e to exit with a code greater than 0 if there was an error
-M do not colorize JSON
The first jq argument is '""' which outputs an empty string, effectively preventing jq from making any processing of the JSON for printing
<<< followed by a string is called a "Here String", and is basically telling bash to treat the string as if it was a file and pass it to STDIN (As input to the command).
Note that you can replace <<<'<MAYBE_INVALID_JSON>' with the path of a file you want to validate and it would still work
Then redirect any STDOUT output (Which normally will contain the JSON) to /dev/null, effectively discarding it
You can also get fancier and output a custom message below the errors by using the following:
jq -reM '""' <<<'<MAYBE_INVALID_JSON>' 1>/dev/null || ( exit_code="$?"; echo 'ERROR: Invalid JSON file. See errors above' 1>&2; exit "$exit_code" )
Where the added portion reads as follows:
|| means: If the previous command failed with a non-zero exit code then run the following command
Open a sub-shell with (. This is actually needed since we want to exit with the original command's exit code
Set the exit_code environment variable to the exit code of the last command $?
Print custom error message using echo
exit the sub-shell with the exit code from the original command $exit_code
Close the sub-shell
For a variable content like for a line, that's been working for me
echo "$my_potential_json"|jq empty > /dev/null 2>&1
if [ $? -eq 0 ]; then
echo "found valid json:$my_potential_json"
do_whatever....
I am trying to build a json with jq with --arg arguments however I'd like for the json not to be able to have a condition if the variable is empty.
An example, if I run the following command
jq -n --arg myvar "${SOMEVAR}" '{ $myvar}'
I'd like the json in that case to be {} if myvar happens to be empty (Because the variable ${SOMEVAR} does not exist) and not { "myvar": "" } which is what I get by just running the command above.
Is there any way to achieve this through some sort of condition?
UPDATE:
Some more details about the use case
I want to build a json based on several environment variables but only include the variables that have a value.
Something like
{"varA": "value", "varB": "value"}
But only include varA if its value is defined and so on. The issue now is that if value is not defined, the property varA will still exist with an empty value and because of the multiple argument/variable nature, using an if/else to build the entire json as suggested will lead to a huge amount of conditions to cover for every possible combination of variables not existing
Suppose you have a template of variable names, in the form of an object as you have suggested you want:
{a, b, c}
Suppose also (for the sake of illustration) that you want to pull in the corresponding values from *ix environment variables. Then you just need to adjust the template, which can be done using this filter:
def adjust: with_entries( env[.key] as $v | select($v != null) | .value = $v );
Example:
Assuming the above filter, together with the following line, is in a file named adjust.jq:
{a,b,c} | adjust
then:
$ export a=123
$ jq -n -f -c adjust.jq
{"a":"123"}
You can use an if/else construct:
jq -n --arg myvar "${SOMEVAR}" 'if ($myvar|length > 0) then {$myvar} else {} end'
It's still not clear where the variable-value pairs are coming from, so maybe it would be simplest to construct the object containing the mapping before invoking jq, and then passing it in using the --argjson or --argfile option?
Hi & thanks in advance.
I'm trying to update a column(version) on an MySQL table from a Bash script.
I've populated a variable with the version numbers, but it fails after applying the first version in the list.
CODE:
UP_VER=`seq ${DB_VER} ${LT_VER} | sed '1d'`
UP_DB=`echo "UPDATE client SET current_db_vers='${UP_VER}' WHERE client_name='${CLIENT}'" | ${MYSQL_ID}`
while read -r line
do
${UP_DB}
if [[ "${OUT}" -eq "0" ]]; then
echo "Database upgraded.."
else
echo "Failed to upgrade.."
exit 1
fi
done < "${UP_VER}"
Thanks
Hopefully solved... My $UP_VER is in a a row not a column.
You're misunderstanding what several shell constructs do:
var=`command` # This executes the command immediately, and stores
# its result (NOT the command itself) in the variable
... < "${UP_VER}" # Treats the contents of $UP_VER as a filename, and tries
# to use that file as input
if [[ "${OUT}" -eq "0" ]]; then # $OUT is not defined anywhere
... current_db_vers='${UP_VER}' ... # this sets current_db_vers to the entire
# list of versions at once
Also, in the shell it's best to use lowercase (or mixed-case) variable names to avoid conflicts with the variables that have special meanings (which are all uppercase).
To fix the first problem, my recommendation is don't try to store shell commands in variables, it doesn't work right. (See BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail!.) Either use a function, or just write the command directly where it's going to be executed. In this case I'd vote for just putting it directly where it's going to be executed. BTW, you're making the same mistake with ${MYSQL_ID}, so I'd recommend fixing that as well.
For the second problem, you can use <<< "${UP_VER}" to feed a variable's contents as input (although this is a bashism, and not available in generic posix shells). But in this case I'd just use a for loop:
for ((ver=db_ver+1; ver<=lt_ver; ver++)); do
For the third problem, the simplest way to test the success of a command is to put it directly in the if:
if somecommand; then
echo "Database upgraded.."
else # ... etc
So, here's my take at a rewrite:
mysql_id() {
# appropriate function definition goes here...
}
for ((ver=db_ver+1; ver<=lt_ver; ver++)); do
if echo "UPDATE client SET current_db_vers='${ver}' WHERE client_name='${client}'" | mysql_id; then
echo "Database upgraded.."
else
echo "Failed to upgrade.."
exit 1
fi
done
... but I'm not sure I understand what it's supposed to do. It seems to be updating current_db_vers one number at a time until it reaches $ver_lt... but why not set it directly to $ver_lt in a single UPDATE?
try something like :
done <<< "${UP_VER}"
I am navigating a Java-based CLI menu on a remote machine with expect inside a bash script and I am trying to extract something from the output without leaving the expect session.
Expect command in my script is:
expect -c "
spawn ssh user#host
expect \"#\"
send \"java cli menu command here\r\"
expect \"java cli prompt\"
send \"java menu command\"
"
###I want to extract a specific string from the above output###
Expect output is:
Id Name
-------------------
abcd 12 John Smith
I want to extract abcd 12 from the above output into another expect variable for further use within the expect script. So that's the 3rd line, first field by using a double-space delimiter. The awk equivalent would be: awk -F ' ' 'NR==3 {$1}'
The big issue is that the environment through which I am navigating with Expect is, as I stated above, a Java CLI based menu so I can't just use awk or anything else that would be available from a bash shell.
Getting out from the Java menu, processing the output and then getting in again is not an option as the login process lasts for 15 seconds so I need to remain inside and extract what I need from the output using expect internal commands only.
You can use regexp in expect itself directly with the use of -re flag. Thanks to Donal on pointing out the single quote and double quote issues. I have given solution using both ways.
I have created a file with the content as follows,
Id Name
-------------------
abcd 12 John Smith
This is nothing but your java program's console output. I have tested this in my system with this. i.e. I just simulated your program's output with cat. You just replace the cat code with your program commands. Simple. :)
Double Quotes :
#!/bin/bash
expect -c "
spawn ssh user#domain
expect \"password\"
send \"mypassword\r\"
expect {\\\$} { puts matched_literal_dollar_sign}
send \"cat input_file\r\"; # Replace this code with your java program commands
expect -re {-\r\n(.*?)\s\s}
set output \$expect_out(1,string)
#puts \$expect_out(1,string)
puts \"Result : \$output\"
"
Single Quotes :
#!/bin/bash
expect -c '
spawn ssh user#domain
expect "password"
send "mypasswordhere\r"
expect "\\\$" { puts matched_literal_dollar_sign}
send "cat input_file\r"; # Replace this code with your java program commands
expect -re {-\r\n(.*?)\s\s}
set output $expect_out(1,string)
#puts $expect_out(1,string)
puts "Result : $output"
'
As you can see, I have used {-\r\n(.*?)\s\s}. Here the braces prevent any variable substitutions. In your output, we have a 2nd line with full of hyphens. Then a newline. Then your 3rd line content. Let's decode the regex used.
-\r\n is to match one literal hyphen and a new line together. This will match the last hyphen in the 2nd line and the newline which in turn make it to 3rd line now. So, .*? will match the required output (i.e. abcd 12) till it encounters double space which is matched by \s\s.
You might be wondering why I need parenthesis which is used to get the sub-match patterns.
In general, expect will save the expect's whole match string in expect_out(0,string) and buffer all the matched/unmatched input to expect_out(buffer). Each sub match will be saved in subsequent numbering of string such as expect_out(1,string), expect_out(2,string) and so on.
As Donal pointed out, it is better to use single quote's approach since it looks less messy. :)
It is not required to escape the \r with the backslash in case of double quotes.
Update :
I have changed the regexp from -\r\n(\w+\s+\w+)\s\s to -\r\n(.*?)\s\s.
With this way - your requirement - such as match any number of letters and single spaces until you encounter first occurrence of double spaces in the output
Now, let's come to your question. You have mentioned that you have tried -\r\n(\w+)\s\s. But, there is a problem here with \w+. Remember \w+ will not match space character. Your output has some spaces in it till double spaces.
The use of regexp will matter based on your requirements on the input string which is going to get matched. You can customize the regular expressions based on your needs.
Update version 2 :
What is the significance of .*?. If you ask separately, I am going to repeat what you commented. In regular expressions, * is a greedy operator and ? is our life saver. Let us consider the string as
Stackoverflow is already overflowing with number of users.
Now, see the effect of the regular expression .*flow as below.
* matches any number of characters. More precisely, it matches the longest string possible while still allowing the pattern itself to match. So, due to this, .* in the pattern matched the characters Stackoverflow is already over and flow in pattern matched the text flow in the string.
Now, in order to prevent the .* to match only up to the first occurrence of the string flow, we are adding the ? to it. It will help the pattern to behave as non-greedy manner.
Now, again coming back to your question. If we have used .*\s\s, then it will match the whole line since it is trying to match as much as possible. This is common behavior of regular expressions.
Update version 3:
Have your code in the following way.
x=$(expect -c "
spawn ssh user#host
expect \"password\"
send \"password\r\"
expect {\\\$} { puts matched_literal_dollar_sign}
send \"cat input\r\"
expect -re {-\r\n(.*?)\s\s}
if {![info exists expect_out(1,string)]} {
puts \"Match did not happen :(\"
exit 1
}
set output \$expect_out(1,string)
#puts \$expect_out(1,string)
puts \"Result : \$output\"
")
y=$?
# $x now contains the output from the 'expect' command, and $y contains the
# exit status
echo $x
echo $y;
If the flow happened properly, then exit code will have value as 0. Else, it will have 1. With this way, you can check the return value in bash script.
Have a look at here to know about the info exists command.