variable expansion in subshell in makefile fails - function

I have a makefile function inside a makefile (myfunction.mk):
.ONESHELL:
define call_script
set +x
mkdir -p $$(dirname $(2))
if [ ! -f $(2) ]; then
echo "" > $(2)
fi
REDIRECT='| tee -a'
echo '>> $(1)'
($(1) ???????? $(2))
RET_CODE=$$?
echo "exit_code is: $$RET_CODE"
if [ ! $$RET_CODE = 0 ]; then
echo "$(3) terminated with error $$RET_CODE"
exit $$RET_CODE
else
if [ ! -z "$(strip $(3))" ]; then
echo "$(3) done"
fi
fi
endef
this function call a script and append result to a log (which is created with its folder if non existing), the result of the script is append only if the makefile variable given as the 4th ($(4)) argument is equal to 'yes'.
you call it like this:
include myfunction.mk
OUTPUT_ENABLED ?= yes
target:
$(call call_script, echo "test", reports/mylog.log, "doing test", OUTPUT_ENABLED)
This works for the most part:
if i replace '????????' by '| tee -a', it works.
if i replace '????????' by $(REDIRECT), it fails.
if i replace '????????' by $$REDIRECT, it fails.
why?
note: running it from a shell /bin/sh: symbolic link to dash
note: of course i want to add a ifeq that allows me to check for $(4) and replace | tee -a by &>>

I'll assume that you use call in a recipe, not flat in your Makefile. There are few problems with your shell script. First, if you try the following on the command line:
mkdir -p reports
REDIRECT='| tee -a'
echo '>> echo "test"'
(echo "test" $REDIRECT reports/mylog.log)
you'll see that echo considers:
"test" $REDIRECT reports/mylog.log
as its arguments. They are expanded and echoed, which prints:
test | tee -a reports/mylog.log
on the standard output, not the effect you expected, I guess. You could, for instance, use eval. On the command line:
eval "echo "test" $REDIRECT reports/mylog.log"
Which, in your Makefile, would become:
eval "$(1) $$REDIRECT $(2)"
Next you should not quote the third parameter of call because the quotes will be passed unmodified and your script will be expanded by make as:
echo " "doing test" terminated with error $RET_CODE"
Again probably not what you want.
Third, you should avoid useless spaces in the parameters of call because they are preserved too (as you can see above between the first 2 double quotes):
.PHONY: foo
foo:
$(call call_script,echo "test",reports/mylog.log,doing test,OUTPUT_ENABLED)
And for your last desired feature, it would be slightly easier to pass the value of OUTPUT_ENABLED to call instead of its name, but let's go this way:
$ cat myfunction.mk
define call_script
set +x
mkdir -p $$(dirname $(2))
if [ ! -f $(2) ]; then
echo "" > $(2)
fi
if [ "$($(4))" = "yes" ]; then
REDIRECT='| tee -a'
else
REDIRECT='&>>'
fi
echo '>> $(1)'
eval "$(1) $$REDIRECT $(2)"
RET_CODE=$$?
echo "exit_code is: $$RET_CODE"
if [ ! $$RET_CODE = 0 ]; then
echo "$(3) terminated with error $$RET_CODE"
exit $$RET_CODE
else
if [ ! -z "$(strip $(3))" ]; then
echo "$(3) done"
fi
fi
endef
$ cat Makefile
.ONESHELL:
include myfunction.mk
OUTPUT_ENABLED ?= yes
target:
$(call call_script,echo "test",reports/mylog.log,doing test,OUTPUT_ENABLED)
Note that I moved the .ONESHELL: in the main Makefile because it is probably better to not hide it inside an included file. Up to you.

The most problematic issue here is that if you pipe your commands, the exit code is the exit code of the last command in a pipe, e.g false | tee foo.log will exit with 0 as tee will most probably succeed. Note also that pipe only redirects stdout, so your log will not contain any stderr messages unless explicitly redirected.
Considering that piping commands influence exit code and lack of portability of $PIPESTATUS (most specifically not being supported in dash), I would try to avoid piping commands and use a temporary file for gathering output, i.e.:
$ cat Makefile
# $(1) - script to execute
# $(2) - log file
# $(3) - description
define call_script
echo '>> $(1)'
$(if $(OUTPUT_ENABLED), \
$(1) > $#.log 2>&1; RET_CODE=$$?; mkdir -p $(dir $(2)); cat $#.log >> $(2); cat $#.log; rm -f $#.log, \
$(1); RET_CODE=$$? \
); \
echo "EXIT_CODE is: $${RET_CODE}"; \
if [ $${RET_CODE} -ne 0 ]; then $(if $(3),echo "$(3) terminated with error $${RET_CODE}";) exit $${RET_CODE}; fi; \
$(if $(3), echo "$(3) done.")
endef
good:
$(call call_script,echo "test",reports/mylog.log,doing test)
bad:
$(call call_script,mkdir /root/foo,reports/mylog.log,intentional fail)
ugly:
$(call call_script,bad_command,reports/mylog.log)
Regular call will not create the logs and will stop on errors:
$ make good bad ugly
echo '>> echo "test"'
>> echo "test"
echo "test"; RET_CODE=$? ; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "doing test terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi; echo "doing test done."
test
EXIT_CODE is: 0
doing test done.
echo '>> mkdir /root/foo'
>> mkdir /root/foo
mkdir /root/foo; RET_CODE=$? ; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "intentional fail terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi; echo "intentional fail done."
mkdir: cannot create directory ‘/root/foo’: Permission denied
EXIT_CODE is: 1
intentional fail terminated with error 1
make: *** [Makefile:19: bad] Error 1
Note that ugly was not built due to failure on bad. Now the same with the log:
$ make good bad ugly OUTPUT_ENABLED=1
echo '>> echo "test"'
>> echo "test"
echo "test" > good.log 2>&1; RET_CODE=$?; mkdir -p reports/; cat good.log >> reports/mylog.log; cat good.log; rm -f good.log; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "doing test terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi; echo "doing test done."
test
EXIT_CODE is: 0
doing test done.
echo '>> mkdir /root/foo'
>> mkdir /root/foo
mkdir /root/foo > bad.log 2>&1; RET_CODE=$?; mkdir -p reports/; cat bad.log >> reports/mylog.log; cat bad.log; rm -f bad.log; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "intentional fail terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi; echo "intentional fail done."
mkdir: cannot create directory ‘/root/foo’: Permission denied
EXIT_CODE is: 1
intentional fail terminated with error 1
make: *** [Makefile:19: bad] Error 1
$ cat reports/mylog.log
test
mkdir: cannot create directory ‘/root/foo’: Permission denied
Note that this time ugly was also not run. But if run later, it will correctly append to the log:
$ make ugly OUTPUT_ENABLED=1
echo '>> bad_command'
>> bad_command
bad_command > ugly.log 2>&1; RET_CODE=$?; mkdir -p reports/; cat ugly.log >> reports/mylog.log; cat ugly.log; rm -f ugly.log; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then exit ${RET_CODE}; fi;
/bin/sh: 1: bad_command: not found
EXIT_CODE is: 127
make: *** [Makefile:22: ugly] Error 127
$ cat reports/mylog.log
test
mkdir: cannot create directory ‘/root/foo’: Permission denied
/bin/sh: 1: bad_command: not found
Personally I am not fan of implementing logging in this way. It is complicated and it only logs output of commands, not make output itself, and only of those commands which are explicitly called to do so. I'd rather keep Makefile clean and simple and just run make 2>&1 | tee log instead to have the output logged.

Related

Shell script: if statement does not work as I want it to

I wrote a shell script (for practice) that should compile a C++ (.cpp) file, automatically generate an executable with clang++ and execute it. My code:
#!/bin/bash
function runcpp() {
CPPFILE=$1
if [ -z $CPPFILE ]; then
echo "You need to specify a path to your .cpp file!"
else
echo -n "Checking if '$CPPFILE' is a valid file..."
if [[ $CPPFILE == "*.cpp" ]]; then
echo -e "\rChecking if '$CPPFILE' is a valid file... successful"
echo -n "Generating executable for '$CPPFILE'..."
clang++ $CPPFILE
echo -e "\rGenerating executable for '$CPPFILE'... done"
fi
fi
}
It's not done yet, however, at line 9 (if [[ $CPPFILE == "*.cpp" ]]; then) something goes wrong: the script exits, even though the file I specified is a .cpp file. My Terminal window:
kali#kali:~$ ls -lha *.cpp
-rw-r--r-- 1 kali kali 98 Feb 9 19:35 test.cpp
kali#kali:~$ runcpp test.cpp
Checking if 'test.cpp' is a valid file...kali#kali:~$

Use arguments as a part of variable inside of function

If I type:
function chk_is_it_started(){
PROCC_NAME_$1="my_process_$1";
echo "PROCC_NAME_$1 is: $PROCC_NAME_$1";
PID_FILE_OF_APP_$1="/run/pidfile_$PROCC_NAME_$1.pid"
PATH_OF_PROCCESS_NAME_$1=`ps -aux|grep $PROCC_NAME_$1|grep -v grep|awk -F" " '{print $12}'`
PID_NUMBER_OF_APP_$1=`ps -aux|grep $PROCC_NAME_$1|grep -v grep|awk -F" " '{print $2}'`
NUMBER_OF_OCCURENCE_$1=`echo ${#PID_NUMBER_OF_APP_$1[#]}`
if [[ "$NUMBER_OF_OCCURENCE_$1" == 0 ]];then
echo -e "Proccess isn't started..\nNow process $PATH_OF_PROCCESS_NAME_$1 is running and I'm creating a PID file..."
python /emu/script/$PROCC_NAME_$1.py & disown & echo $! > $PID_FILE_OF_APP_$1
else
echo "Proccess is STARTRED"
fi
}
chk_is_it_started blabla;
I will got the error:
root#orangepipc:~# chk_is_it_started blabla;
Could not find the database of available applications, run update-command-not-found as root to fix this
PROCC_NAME_blabla=my_process_blabla: command not found
PROCC_NAME_blabla is: blabla
-bash: PID_FILE_OF_APP_blabla=/run/pidfile_blabla.pid: No such file or directory
Could not find the database of available applications, run update-command-not-found as root to fix this
PATH_OF_PROCCESS_NAME_blabla=: command not found
Could not find the database of available applications, run update-command-not-found as root to fix this
PID_NUMBER_OF_APP_blabla=: command not found
-bash: ${#PID_NUMBER_OF_APP_$1[#]}: bad substitution
Could not find the database of available applications, run update-command-not-found as root to fix this
NUMBER_OF_OCCURENCE_blabla=: command not found
Proccess is STARTRED
But it is not!
Where I'm Making the misstake?
If I'm using th ecode without function it work!
Thx
I found the solution...
function chk_is_it_started(){
PROCC_NAME="dht22_$1"
# echo "PROCC_NAME_$1 is: $PROCC_NAME"
PID_FILE_OF_APP="/run/pidfile_$PROCC_NAME.pid"
# echo "PID_FILE_OF_APP is: $PID_FILE_OF_APP"
PATH_OF_PROCCESS_NAME=`ps -aux|grep $PROCC_NAME_$1|grep -v grep|awk -F" " '{print $12}'`
# echo "PATH_OF_PROCCESS_NAME is: $PATH_OF_PROCCESS_NAME"
PID_NUMBER_OF_APP=`ps -aux|grep $PROCC_NAME_$1|grep -v grep|awk -F" " '{print $2}'`
# echo "PID_NUMBER_OF_APP is $PID_NUMBER_OF_APP"
PID_NUMBER_OF_APP=( $PID_NUMBER_OF_APP )
# echo "PID_NUMBER_OF_APP is $PID_NUMBER_OF_APP"
NUMBER_OF_OCCURENCE=`echo ${#PID_NUMBER_OF_APP[#]}`
# echo "NUMBER_OF_OCCURENCE is: $NUMBER_OF_OCCURENCE"
if [[ "$NUMBER_OF_OCCURENCE" == 0 ]];then
echo -e "Proccess isn't started..\nNow process $PATH_OF_PROCCESS_NAME is running and I create a PID file..."
python /emu/script/$PROCC_NAME.py & disown & echo $! > $PID_FILE_OF_APP
# exit
else
echo "Proccess is STARTRED"
fi
if [[ "$NUMBER_OF_OCCURENCE" > 1 ]];then
echo -e "Process $PROCC_NAME.py is started more than 1x"
echo -e "Now killing all proccess one by one"
while [ "$NUMBER_OF_OCCURENCE" != "1" ];
do
echo "Usao sam u while"
PID_NUMBER_OF_APP=`ps -aux|grep $PROCC_NAME|grep -v grep|awk -F" " '{print $2}'`
echo "PID_NUMBER_OF_APP is: $PID_NUMBER_OF_APP"
PID_NUMBER_OF_APP=( $PID_NUMBER_OF_APP )
echo "PID_NUMBER_OF_APP is $PID_NUMBER_OF_APP"
NUMBER_OF_OCCURENCE=`echo ${#PID_NUMBER_OF_APP[#]}`
echo "NUMBER_OF_OCCURENCE is: $NUMBER_OF_OCCURENCE"
kill $PID_NUMBER_OF_APP
rm -fr $PID_FILE_OF_APP
done
echo -e "Starting process $PROCC_NAME.py and creating a PID file..."
python /emu/script/$PROCC_NAME.py & echo $! > $PID_FILE_OF_APP
fi
}
chk_is_it_started bla1
chk_is_it_started bla2
Btw saluting user who gave vote -1 to my question :)

How can I execute MySQL commands line by line from bash and capture the output?

If there is an alternative to bash, I will appreciate it too.
I have a large dump of MySQL commands (over 10 GB)
When restoring the dump I get a few warnings and occasionally an error. I need to execute those commands and process all warnings and errors. And, preferibly to do it automatically.
mysql --show-warnings
tee logfile.log
source dump.sql
The logfile will contain many lines telling each command was successful, and will display some warnings, particulartly truncate colums. But the original file has tens of thousands of very large INSERTs, the log is not particularly helpful. Despite it requires some kind of supervised interaction. (I cannot program a crontab, for example.)
#!/bin/bash
echo "tee logfile.log" > script.sql
echo "source $1" > script.sql
mysql --show-warnings < script.sql > tmpfile.log 2>&1
cat tmpfile.log >> logfile.log
The tee command doesn't work in this batch environment. I can capture all the warnings, but I cannot figure out which command produced each warning.
So I came down with this small monstruosity:
#!/bin/bash
ERRFILE=$(basename "$0" .sh).err.log
LOGFILE=$(basename "$1" .sql).log
log_action() {
WARN=$(cat)
[ -z "$WARN" ] || echo -e "Line ${1}: ${WARN}\n${2}" >> "$LOGFILE"
}
echo 0 > "$ERRFILE"
log_error() {
ERNO=$(cat "$ERRFILE")
ERR=$(cat)
[ -z "$ERR" ] || echo -e "*** ERROR ***\nLine ${1}: ${ERR}\n${2}" >> "$LOGFILE"
(( ERNO++ ))
echo $ERNO > "$ERRFILE"
}
COUNT=0
COMMAND=''
echo -e "**** BEGIN $(date +%Y-%m-%d\ %H:%M:%S)\n" > "$LOGFILE"
exec 4> >(log_action $COUNT "$COMMAND")
exec 5> >(log_error $COUNT "$COMMAND")
exec 3> >(mysql --show-warnings >&4 2>&5)
while IFS='' read -r LINE || [[ -n "$line" ]]
do
(( COUNT++ ))
[ ${#LINE} -eq 0 ] && continue # discard blank lines
[ "${LINE:0:2}" = "--" ] && continue # discard comments
COMMAND+="$LINE" # build command
[ "${LINE: -1}" != ";" ] && continue # if not finnished keep building
echo $COMMAND >&3 # otherwise execute
COMMAND=''
done < "$1"
exec 3>$-
exec 5>$-
exec 4>$-
echo -e "**** END $(date +%Y-%m-%d\ %H:%M:%S)\n" >> "$LOGFILE"
ERRS=$(cat "$ERRFILE")
[ "ERRS" = 0 ] || echo "${ERRS} Errors." >&2
This scans the file at $1 and sends the commands to an open MySQL connection at &3. That part is working fine.
The capture of warnings and errors is not working though.
It only records the first error.
It only records the first warning.
I haven't find a good way to pass the line number $COUNT and offending command $COMMAND to the recording functions.
The only error is after the time stamps, and the only warning is after the error, which is not the chronology of the script.

gnu parallel not recognize user-defined functions

I cannot get the gnu parallel function to implement a custom function that I built.
My function is:
function run_cuffLinks() {
inputBAM="${HOME}/Analyses/P_miniata/CleanUpPipeline/TH_${1}/${1}.realigned.bam"
if [[ ! -f $inputBAM ]]; then echo -e "$inputBAM could not be found\nexit 1" ; fi
WORKING_DIR="${HOME}/data/CuffLinks/TH_$1"
if [[ ! -d $WORKING_DIR ]]; then mkdir -p $WORKING_DIR; fi
REF="${HOME}/ReferenceSequences/GATK_pmin.scaf.fa"
if [[ ! -f $REF ]]; then echo -e "$inputBAM could not be found\nexit 1" ; exit 1; fi
GTF_FILE="${HOME}/ReferenceSequences/genes.sorted.gff3"
if [[ ! -f $GTF_FILE ]]; then echo -e "$inputBAM could not be found\nexit 1" ; exit 1; fi
cufflinks \
--output-dir $WORKING_DIR \
--num-threads 2 \
--frag-len-mean 100 \
--GTF-guide $GTF_FILE \
--frag-bias-correct $REF \
-L "HH" \
$inputBAM ;
}
When I enter:
parallel --no-notice -j+2 run_cuffLinks {} ::: sample1 sample2 sample3
I get the output:
/bin/bash: run_cuffLinks: command not found
/bin/bash: run_cuffLinks: command not found
/bin/bash: run_cuffLinks: command not found
If I include a '$' symbol in front of the function name, I get:
/bin/bash: sample1: command not found
/bin/bash: sample2: command not found
/bin/bash: sample3: command not found
I have also tried using the -pipe --recend and --rrs options, but without a positive result.
Is GNU parallel not able to process user-defined functions?
You do not write whether you have walked through the tutorial (man parallel_tutorial). In that it shows that you must export -f the function, and since you do not write that, I believe you might have forgotten that:
export -f run_cuffLinks
parallel ...
Since version 20180522 you can also use env_parallel:
env_parallel --session
[define functions and variables here that you want parallel to see]
# Use env_parallel like you would parallel
env_parallel run_cuffLinks ...
PS: Use --bibtex once to avoid --no-notice in the future.

What's a Good Way to Commit All MySQL Settings Needed to Get a Django App Running?

I'm in the middle of making my first django app, and I'd like to commit it to git in such a way that someone can clone it down and start working on it with the least amount of trouble. One of the things I needed to do to get things up and running was to create a new db in my local mysql installation and create a new user there. I'd love to let someone clone things down and have that done automatically for them. Is there a good way to do this?
Use mysql-python and write a python script to create the database or alter my shell script to make it suit your needs.
#!/bin/bash
function pre_checks() {
if [[ "$1" -ne 3 ]]; then
echo "Usage: $0 [DATABASE NAME] [USERNAME] [HOST]"
return 1
fi
if ! command -v /usr/bin/mysql >/dev/null 2>&1; then
echo "Mysql is not installed."
return 1
fi
echo -n "Create the database '${2}' and the user '${3}' now? (y/n) "
read ANSWER
case "$ANSWER" in
"y"|"Y")
echo -n "Password for ${3}: "
read -s USER_PW
echo
return 0 ;;
"n"|"N"| *)
echo "Bye."
return 1 ;;
esac
}
function create_db() {
Q1="CREATE DATABASE IF NOT EXISTS ${1} CHARACTER SET utf8;"
Q2="GRANT ALL ON *.* TO '${2}'#'${3}' IDENTIFIED BY '$USER_PW';"
Q3="FLUSH PRIVILEGES;"
Q4="SHOW DATABASES;"
SQL="${Q1} ${Q2} ${Q3} ${Q4}"
echo "Query:"
echo "${SQL}"
echo -n "Run query now? (y/n) "
read ANSWER
case "$ANSWER" in
"y" | "Y" )
/usr/bin/mysql -uroot -p -e "$SQL" || echo "Failure."
;;
"n" | "N" | *)
echo "Bye."
return 1
;;
esac
}
pre_checks "$#" "$1" "$2" && create_db "$1" "$2" "$3"