I have a pretty simple script that is something like the following:
#!/bin/bash
VAR1="$1"
MOREF='sudo run command against $VAR1 | grep name | cut -c7-'
echo $MOREF
When I run this script from the command line and pass it the arguments, I am not getting any output. However, when I run the commands contained within the $MOREF variable, I am able to get output.
How can one take the results of a command that needs to be run within a script, save it to a variable, and then output that variable on the screen?
In addition to backticks `command`, command substitution can be done with $(command) or "$(command)", which I find easier to read, and allows for nesting.
OUTPUT=$(ls -1)
echo "${OUTPUT}"
MULTILINE=$(ls \
-1)
echo "${MULTILINE}"
Quoting (") does matter to preserve multi-line variable values; it is optional on the right-hand side of an assignment, as word splitting is not performed, so OUTPUT=$(ls -1) would work fine.
$(sudo run command)
If you're going to use an apostrophe, you need `, not '. This character is called "backticks" (or "grave accent"):
#!/bin/bash
VAR1="$1"
VAR2="$2"
MOREF=`sudo run command against "$VAR1" | grep name | cut -c7-`
echo "$MOREF"
Some Bash tricks I use to set variables from commands
Sorry, there is a loong answer, but as bash is a shell, where the main goal is to run other unix commands and react on result code and/or output, ( commands are often piped filter, etc... ).
Storing command output in variables is something basic and fundamental.
Therefore, depending on
compatibility (posix)
kind of output (filter(s))
number of variable to set (split or interpret)
execution time (monitoring)
error trapping
repeatability of request (see long running background process, further)
interactivity (considering user input while reading from another input file descriptor)
do I miss something?
First simple, old (obsolete), and compatible way
myPi=`echo '4*a(1)' | bc -l`
echo $myPi
3.14159265358979323844
Compatible, second way
As nesting could become heavy, parenthesis was implemented for this
myPi=$(bc -l <<<'4*a(1)')
Using backticks in script is to be avoided today.
Nested sample:
SysStarted=$(date -d "$(ps ho lstart 1)" +%s)
echo $SysStarted
1480656334
bash features
Reading more than one variable (with Bashisms)
df -k /
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/dm-0 999320 529020 401488 57% /
If I just want a used value:
array=($(df -k /))
you could see an array variable:
declare -p array
declare -a array='([0]="Filesystem" [1]="1K-blocks" [2]="Used" [3]="Available" [
4]="Use%" [5]="Mounted" [6]="on" [7]="/dev/dm-0" [8]="999320" [9]="529020" [10]=
"401488" [11]="57%" [12]="/")'
Then:
echo ${array[9]}
529020
But I often use this:
{ read -r _;read -r filesystem size using avail prct mountpoint ; } < <(df -k /)
echo $using
529020
( The first read _ will just drop header line. ) Here, in only one command, you will populate 6 different variables (shown by alphabetical order):
declare -p avail filesystem mountpoint prct size using
declare -- avail="401488"
declare -- filesystem="/dev/dm-0"
declare -- mountpoint="/"
declare -- prct="57%"
declare -- size="999320"
declare -- using="529020"
Or
{ read -a head;varnames=(${head[#]//[K1% -]});varnames=(${head[#]//[K1% -]});
read ${varnames[#],,} ; } < <(LANG=C df -k /)
Then:
declare -p varnames ${varnames[#],,}
declare -a varnames=([0]="Filesystem" [1]="blocks" [2]="Used" [3]="Available" [4]="Use" [5]="Mounted" [6]="on")
declare -- filesystem="/dev/dm-0"
declare -- blocks="999320"
declare -- used="529020"
declare -- available="401488"
declare -- use="57%"
declare -- mounted="/"
declare -- on=""
Or even:
{ read _ ; read filesystem dsk[{6,2,9}] prct mountpoint ; } < <(df -k /)
declare -p mountpoint dsk
declare -- mountpoint="/"
declare -a dsk=([2]="529020" [6]="999320" [9]="401488")
(Note Used and Blocks is switched there: read ... dsk[6] dsk[2] dsk[9] ...)
... will work with associative arrays too: read _ disk[total] disk[used] ...
Other related sample: Parsing xrandr output: and end of Firefox tab by bash in a size of x% of display size? or at AskUbuntu.com Parsing xrandr output
Dedicated fd using unnamed fifo:
There is an elegent way! In this sample, I will read /etc/passwd file:
users=()
while IFS=: read -u $list user pass uid gid name home bin ;do
((uid>=500)) &&
printf -v users[uid] "%11d %7d %-20s %s\n" $uid $gid $user $home
done {list}</etc/passwd
Using this way (... read -u $list; ... {list}<inputfile) leave STDIN free for other purposes, like user interaction.
Then
echo -n "${users[#]}"
1000 1000 user /home/user
...
65534 65534 nobody /nonexistent
and
echo ${!users[#]}
1000 ... 65534
echo -n "${users[1000]}"
1000 1000 user /home/user
This could be used with static files or even /dev/tcp/xx.xx.xx.xx/yyy with x for ip address or hostname and y for port number or with the output of a command:
{
read -u $list -a head # read header in array `head`
varnames=(${head[#]//[K1% -]}) # drop illegal chars for variable names
while read -u $list ${varnames[#],,} ;do
((pct=available*100/(available+used),pct<10)) &&
printf "WARN: FS: %-20s on %-14s %3d <10 (Total: %11u, Use: %7s)\n" \
"${filesystem#*/mapper/}" "$mounted" $pct $blocks "$use"
done
} {list}< <(LANG=C df -k)
And of course with inline documents:
while IFS=\; read -u $list -a myvar ;do
echo ${myvar[2]}
done {list}<<"eof"
foo;bar;baz
alice;bob;charlie
$cherry;$strawberry;$memberberries
eof
Practical sample parsing CSV files:
As this answer is loong enough, for this paragraph,
I just will let you refer to
this answer to How to parse a CSV file in Bash?, I read a file by using an unnamed fifo, using syntax like:
exec {FD}<"$file" # open unnamed fifo for read
IFS=';' read -ru $FD -a headline
while IFS=';' read -ru $FD -a row ;do ...
... But using bash loadable CSV module.
On my website, you may find the same script, reading CSV as inline document.
Sample function for populating some variables:
#!/bin/bash
declare free=0 total=0 used=0 mpnt='??'
getDiskStat() {
{
read _
read _ total used free _ mpnt
} < <(
df -k ${1:-/}
)
}
getDiskStat $1
echo "$mpnt: Tot:$total, used: $used, free: $free."
Nota: declare line is not required, just for readability.
About sudo cmd | grep ... | cut ...
shell=$(cat /etc/passwd | grep $USER | cut -d : -f 7)
echo $shell
/bin/bash
(Please avoid useless cat! So this is just one fork less:
shell=$(grep $USER </etc/passwd | cut -d : -f 7)
All pipes (|) implies forks. Where another process have to be run, accessing disk, libraries calls and so on.
So using sed for sample, will limit subprocess to only one fork:
shell=$(sed </etc/passwd "s/^$USER:.*://p;d")
echo $shell
And with Bashisms:
But for many actions, mostly on small files, Bash could do the job itself:
while IFS=: read -a line ; do
[ "$line" = "$USER" ] && shell=${line[6]}
done </etc/passwd
echo $shell
/bin/bash
or
while IFS=: read loginname encpass uid gid fullname home shell;do
[ "$loginname" = "$USER" ] && break
done </etc/passwd
echo $shell $loginname ...
Going further about variable splitting...
Have a look at my answer to How do I split a string on a delimiter in Bash?
Alternative: reducing forks by using backgrounded long-running tasks
In order to prevent multiple forks like
myPi=$(bc -l <<<'4*a(1)'
myRay=12
myCirc=$(bc -l <<<" 2 * $myPi * $myRay ")
or
myStarted=$(date -d "$(ps ho lstart 1)" +%s)
mySessStart=$(date -d "$(ps ho lstart $$)" +%s)
This work fine, but running many forks is heavy and slow.
And commands like date and bc could make many operations, line by line!!
See:
bc -l <<<$'3*4\n5*6'
12
30
date -f - +%s < <(ps ho lstart 1 $$)
1516030449
1517853288
So we could use a long running background process to make many jobs, without having to initiate a new fork for each request.
You could have a look how reducing forks make Mandelbrot bash, improve from more than eight hours to less than 5 seconds.
Under bash, there is a built-in function: coproc:
coproc bc -l
echo 4*3 >&${COPROC[1]}
read -u $COPROC answer
echo $answer
12
echo >&${COPROC[1]} 'pi=4*a(1)'
ray=42.0
printf >&${COPROC[1]} '2*pi*%s\n' $ray
read -u $COPROC answer
echo $answer
263.89378290154263202896
printf >&${COPROC[1]} 'pi*%s^2\n' $ray
read -u $COPROC answer
echo $answer
5541.76944093239527260816
As bc is ready, running in background and I/O are ready too, there is no delay, nothing to load, open, close, before or after operation. Only the operation himself! This become a lot quicker than having to fork to bc for each operation!
Border effect: While bc stay running, they will hold all registers, so some variables or functions could be defined at initialisation step, as first write to ${COPROC[1]}, just after starting the task (via coproc).
Into a function newConnector
You may found my newConnector function on GitHub.Com or on my own site (Note on GitHub: there are two files on my site. Function and demo are bundled into one unique file which could be sourced for use or just run for demo.)
Sample:
source shell_connector.sh
tty
/dev/pts/20
ps --tty pts/20 fw
PID TTY STAT TIME COMMAND
29019 pts/20 Ss 0:00 bash
30745 pts/20 R+ 0:00 \_ ps --tty pts/20 fw
newConnector /usr/bin/bc "-l" '3*4' 12
ps --tty pts/20 fw
PID TTY STAT TIME COMMAND
29019 pts/20 Ss 0:00 bash
30944 pts/20 S 0:00 \_ /usr/bin/bc -l
30952 pts/20 R+ 0:00 \_ ps --tty pts/20 fw
declare -p PI
bash: declare: PI: not found
myBc '4*a(1)' PI
declare -p PI
declare -- PI="3.14159265358979323844"
The function myBc lets you use the background task with simple syntax.
Then for date:
newConnector /bin/date '-f - +%s' #0 0
myDate '2000-01-01'
946681200
myDate "$(ps ho lstart 1)" boottime
myDate now now
read utm idl </proc/uptime
myBc "$now-$boottime" uptime
printf "%s\n" ${utm%%.*} $uptime
42134906
42134906
ps --tty pts/20 fw
PID TTY STAT TIME COMMAND
29019 pts/20 Ss 0:00 bash
30944 pts/20 S 0:00 \_ /usr/bin/bc -l
32615 pts/20 S 0:00 \_ /bin/date -f - +%s
3162 pts/20 R+ 0:00 \_ ps --tty pts/20 fw
From there, if you want to end one of background processes, you just have to close its fd:
eval "exec $DATEOUT>&-"
eval "exec $DATEIN>&-"
ps --tty pts/20 fw
PID TTY STAT TIME COMMAND
4936 pts/20 Ss 0:00 bash
5256 pts/20 S 0:00 \_ /usr/bin/bc -l
6358 pts/20 R+ 0:00 \_ ps --tty pts/20 fw
which is not needed, because all fd close when the main process finishes.
As they have already indicated to you, you should use `backticks`.
The alternative proposed $(command) works as well, and it also easier to read, but note that it is valid only with Bash or KornShell (and shells derived from those),
so if your scripts have to be really portable on various Unix systems, you should prefer the old backticks notation.
I know three ways to do it:
Functions are suitable for such tasks:**
func (){
ls -l
}
Invoke it by saying func.
Also another suitable solution could be eval:
var="ls -l"
eval $var
The third one is using variables directly:
var=$(ls -l)
OR
var=`ls -l`
You can get the output of the third solution in a good way:
echo "$var"
And also in a nasty way:
echo $var
Just to be different:
MOREF=$(sudo run command against $VAR1 | grep name | cut -c7-)
When setting a variable make sure you have no spaces before and/or after the = sign. I literally spent an hour trying to figure this out, trying all kinds of solutions! This is not cool.
Correct:
WTFF=`echo "stuff"`
echo "Example: $WTFF"
Will Fail with error "stuff: not found" or similar
WTFF= `echo "stuff"`
echo "Example: $WTFF"
If you want to do it with multiline/multiple command/s then you can do this:
output=$( bash <<EOF
# Multiline/multiple command/s
EOF
)
Or:
output=$(
# Multiline/multiple command/s
)
Example:
#!/bin/bash
output="$( bash <<EOF
echo first
echo second
echo third
EOF
)"
echo "$output"
Output:
first
second
third
Using heredoc, you can simplify things pretty easily by breaking down your long single line code into a multiline one. Another example:
output="$( ssh -p $port $user#$domain <<EOF
# Breakdown your long ssh command into multiline here.
EOF
)"
You need to use either
$(command-here)
or
`command-here`
Example
#!/bin/bash
VAR1="$1"
VAR2="$2"
MOREF="$(sudo run command against "$VAR1" | grep name | cut -c7-)"
echo "$MOREF"
If the command that you are trying to execute fails, it would write the output onto the error stream and would then be printed out to the console.
To avoid it, you must redirect the error stream:
result=$(ls -l something_that_does_not_exist 2>&1)
This is another way and is good to use with some text editors that are unable to correctly highlight every intricate code you create:
read -r -d '' str < <(cat somefile.txt)
echo "${#str}"
echo "$str"
You can use backticks (also known as accent graves) or $().
Like:
OUTPUT=$(x+2);
OUTPUT=`x+2`;
Both have the same effect. But OUTPUT=$(x+2) is more readable and the latest one.
Here are two more ways:
Please keep in mind that space is very important in Bash. So, if you want your command to run, use as is without introducing any more spaces.
The following assigns harshil to L and then prints it
L=$"harshil"
echo "$L"
The following assigns the output of the command tr to L2. tr is being operated on another variable, L1.
L2=$(echo "$L1" | tr [:upper:] [:lower:])
Mac/OSX nowadays come with old Bash versions, ie GNU bash, version 3.2.57(1)-release (arm64-apple-darwin21). In this case, one can use:
new_variable="$(some_command)"
A concrete example:
newvar="$(echo $var | tr -d '123')"
Note the (), instead of the usual {} in Bash 4.
Some may find this useful.
Integer values in variable substitution, where the trick is using $(()) double brackets:
N=3
M=3
COUNT=$N-1
ARR[0]=3
ARR[1]=2
ARR[2]=4
ARR[3]=1
while (( COUNT < ${#ARR[#]} ))
do
ARR[$COUNT]=$((ARR[COUNT]*M))
(( COUNT=$COUNT+$N ))
done
When I try to execute below command it is showing Error like "extra characters after close-quote" but I gave it properly & when i try to it in unix command line terminal is opening properly.
exec gnome-terminal -e 'sh -c "bsub -Ip -n 1 -M <Memory> -q <queue_name> make"'
Can any one help me to resolve this issue or is there any way to do the same thing ??
Edited -> changed " from before sh to before bsub
Tcl's quoting is not the shell's quoting. Tcl uses {…} like the shell uses single quotes, except that braces nest nicely. Nesting single quotes is a recipe for shell headaches.
exec gnome-terminal -e {sh -c "bsub -Ip -n 1 -M <Memory> -q <queue_name> make"}
However, in this case I'd instead be tempted to go with this:
set memory "<Memory>"
set queue "<queue_name>"
set command "make"
set bsubcmd "bsub -Ip -n 1 -M $memory -q $queue $command"
# It's much more convenient to build this command like this here.
# Otherwise you're doing lots of backslashes and so on and it's horrible and very easy to make bugs
exec gnome-terminal -e [format {sh -c "%s"} $bsubcmd]
The only really messy thing is that command and bsubcmd have to be built using shell syntax if you're passing spaces around. “Fortunately” you're dealing with make anyway, so you probably really want to avoid having spaces in names passed there.
I run Modelsim in the cmd from a python program.
I use the following code which call a tcl script which run the modelsim:
os.system("vsim -c -do top_tb_simulate_reg.tcl " )
The tcl script contain the following:
vsim -voptargs="+acc" +UVM_TESTNAME=test_name +UVM_MAX_QUIT_COUNT=1 +UVM_VERBOSITY=UVM_LOW \
-t 1ps -L unisims_verm -L generic_baseblocks_v2_1_0 -L axi_infrastructure_v1_1_0 \
-L dds_compiler_v6_0_12 -lib xil_defaultlib xil_defaultlib.girobo2_tb_top \
xil_defaultlib.glbl
I want that the value of the +UVM_TESTNAME will be an argument which I passed from the cmd when I execute:
os.system("vsim -c -do top_tb_simulate_reg.tcl " )
How can I do it?
I tried the following with no succees:
Python script:
os.system("vsim -c -do top_tb_simulate_reg.tcl axi_rd_only_test" )
Simulation file (tcl script)
vsim -voptargs="+acc" +UVM_TESTNAME=$argv +UVM_MAX_QUIT_COUNT=1 +UVM_VERBOSITY=UVM_LOW \
-t 1ps -L unisims_verm -L generic_baseblocks_v2_1_0 -L axi_infrastructure_v1_1_0 \
-L dds_compiler_v6_0_12 -lib xil_defaultlib xil_defaultlib.girobo2_tb_top \
xil_defaultlib.glbl
I got the following error:
# ** Error: (vsim-3170) Could not find 'C:/raft/raftortwo/girobo2/ver/sim/work.axi_rd_only_test'.
The problem is that the vsim binary is doing its own processing of the arguments, and that is interfering. While yes, you can probably find a way around this by reading the vsim documentation, the simplest way around this is to pass values via environment variables. They're inherited by a process from its parent process, and are fine for passing most things. (The exception are security tokens, which should always be passed in files with correctly-set permissions, rather than either environment variables or command-line arguments.)
In your python code:
# Store the value in the *inheritable* environment
os.environ["MY_TEST_CASE"] = "axi_rd_only_test"
# Do the call; the environment gets passed over behind the scenes
os.system("vsim -c -do top_tb_simulate_reg.tcl " )
In your tcl code:
# Read out of the inherited environment
set name $env(MY_TEST_CASE)
# Use it! (Could do this as one line, but that's hard to read)
vsim -voptargs="+acc" +UVM_TESTNAME=$name +UVM_MAX_QUIT_COUNT=1 +UVM_VERBOSITY=UVM_LOW \
-t 1ps -L unisims_verm -L generic_baseblocks_v2_1_0 -L axi_infrastructure_v1_1_0 \
-L dds_compiler_v6_0_12 -lib xil_defaultlib xil_defaultlib.girobo2_tb_top \
xil_defaultlib.glbl
Late to the party but I found a great workaround for your obstacle. The do command within Modelsim's TCL instance does accept parameters. See command reference.
vsim -c -do filename.tcl can't take parameters, but you can use vsim -c -do "do filename.tcl params".
In your case this translates to os.system('vsim -c -do "do top_tb_simulate_reg.tcl axi_rd_only_test"'). Your .tcl script will find the parameter passed through the variable $1.
I hope to helps anyone!
I am creating a function (below) with which you can provide an argument, a directory. I test if the $argv is a directory with -d option, but that doesn’t seem to work, it always return true even if no arguments are supplied. I also tried test -n $argv -a -d $argv to test is $argv is empty sting, but that returns test: Missing argument at index 1 error. How should I test if any argument is provided with the function or not? Why is test -d $argv not working, from my understanding it should be false when no argument is provided, because empty string is not a directory.
function fcd
if test -d $argv
open $argv
else
open $PWD
end
end
Thanks for the help.
count is the right way to do this. For the common case of checking whether there are any arguments, you can use its exit status:
function fcd
if count $argv > /dev/null
open $argv
else
open $PWD
end
end
To answer your second question, test -d $argv returns true if $argv is empty, because POSIX requires that when test is passed one argument, it must "Exit true (0) if $1 is not null; otherwise, exit false". So when $argv is empty, test -d $argv means test -d which must exit true because -d is not empty! Argh!
edit Added a missing end, thanks to Ismail for noticing
In fish 2.1+ at least, you can name your arguments, which then allows for arguably more semantic code:
function fcd --argument-names 'filename'
if test -n "$filename"
open $filename
else
open $PWD
end
end
if not set -q argv[1]
echo 'none'
else
echo 'yes'
end
From the man set page:
set ( -q | --query ) [SCOPE_OPTIONS] VARIABLE_NAMES...
o -q or --query test if the specified variable names are defined. Does not output anything, but the builtins exit status is the number of variables specified
that were not defined.
$argv is a list, so you want to look at the first element, if there are elements in that list:
if begin; test (count $argv) -gt 0; and test -d $argv[1]; end
open $argv[1]
else
open $PWD
end
Maybe is non related, but i would like to add another perspective for the question.
I want to broaden the insight to a wider scope the scope of testing the shell code with the libraries developed in the fisherman group.
With mock you can check if the open command is called safely without side effects.
Example.:
function fcd
if count $argv > /dev/null
open $argv
else
open $PWD
end
end
mock open 0 "echo \$args"
fcd "cool" # echoes cool
mock open 0 "echo \$args"
fcd # echoes $PWD
Is a recent library, but it could help to test things that might be dangerous, like for example rm
mock rm 0 "echo nothing would happen on \$args"
rm "some file" # simply echoes the message with a list of the files that would been affected
I hope that it gives a more out of the box point of view
P.S.: Sorry for the blatant publicity, but i think that is a cool idea that would be nice to be adopted by shell scripters, to test and add sustainability to shell scripts is nuts. :P
EDIT: i recently noticed a bug about the sample i posted. Please do not use rm *, because the asterisk is not treated as a param, instead fish shell expands asterisk to the list of files found and the command only mocks the first call, this means that the first file would be ignored by the mock, but all the subsequent files will get erased so please be careful if trying the sample and use a single file for the example not the wildcard.
I'm having an issue passing variables to a Bash script using QSub.
Assume I have a Bash script named example. The format of example is the following:
#!/bin/bash
# (assume other variables have been set)
echo $1 $2 $3 $4
So, executing "bash example.sh this is a test" on Terminal (I am using Ubuntu 12.04.3 LTS, if that helps) produces the output "this is a test".
However, when I enter "qsub -v this,is,a,test example.sh", I get no output. I checked the output file that QSub produces, but the line "this is a test" is nowhere to be found.
Any help would be appreciated.
Thank you.
Using PBSPro or SGE, arguments can simply be placed after the script name as may seem intuitive.
qsub example.sh hello world
In Torque, command line arguments can be submitted using the -F option. Your example.sh will look something like this:
#!/bin/bash
echo "$1 $2"
and your command like so:
qsub -F "hello world" example.sh
Alternatively, environment variables can be set using -v with a comma-separated list of variables.
#!/bin/bash
echo "$FOO $BAR"
and your command like so:
qsub -v FOO="hello",BAR="world" example.sh
(This may be better phrased as a comment on #William Hay's answer, but I don't have the reputation to do so.)
Not sure which batch scheduler you are using but on PBSPro or SGE then submitting with qsub example.sh this is a test should do what you want.
The Torque batch scheduler doesn't (AFAIK) allow passing command line arguments to the script this way. You would need to create a script looking something like this.
#!/bin/bash
echo $FOO
Then submit it with a command like:
qsub -v FOO="This is a test" example.sh