I have a makefile function inside a makefile (myfunction.mk):
.ONESHELL:
define call_script
set +x
mkdir -p $$(dirname $(2))
if [ ! -f $(2) ]; then
echo "" > $(2)
fi
REDIRECT='| tee -a'
echo '>> $(1)'
($(1) ???????? $(2))
RET_CODE=$$?
echo "exit_code is: $$RET_CODE"
if [ ! $$RET_CODE = 0 ]; then
echo "$(3) terminated with error $$RET_CODE"
exit $$RET_CODE
else
if [ ! -z "$(strip $(3))" ]; then
echo "$(3) done"
fi
fi
endef
this function call a script and append result to a log (which is created with its folder if non existing), the result of the script is append only if the makefile variable given as the 4th ($(4)) argument is equal to 'yes'.
you call it like this:
include myfunction.mk
OUTPUT_ENABLED ?= yes
target:
$(call call_script, echo "test", reports/mylog.log, "doing test", OUTPUT_ENABLED)
This works for the most part:
if i replace '????????' by '| tee -a', it works.
if i replace '????????' by $(REDIRECT), it fails.
if i replace '????????' by $$REDIRECT, it fails.
why?
note: running it from a shell /bin/sh: symbolic link to dash
note: of course i want to add a ifeq that allows me to check for $(4) and replace | tee -a by &>>
I'll assume that you use call in a recipe, not flat in your Makefile. There are few problems with your shell script. First, if you try the following on the command line:
mkdir -p reports
REDIRECT='| tee -a'
echo '>> echo "test"'
(echo "test" $REDIRECT reports/mylog.log)
you'll see that echo considers:
"test" $REDIRECT reports/mylog.log
as its arguments. They are expanded and echoed, which prints:
test | tee -a reports/mylog.log
on the standard output, not the effect you expected, I guess. You could, for instance, use eval. On the command line:
eval "echo "test" $REDIRECT reports/mylog.log"
Which, in your Makefile, would become:
eval "$(1) $$REDIRECT $(2)"
Next you should not quote the third parameter of call because the quotes will be passed unmodified and your script will be expanded by make as:
echo " "doing test" terminated with error $RET_CODE"
Again probably not what you want.
Third, you should avoid useless spaces in the parameters of call because they are preserved too (as you can see above between the first 2 double quotes):
.PHONY: foo
foo:
$(call call_script,echo "test",reports/mylog.log,doing test,OUTPUT_ENABLED)
And for your last desired feature, it would be slightly easier to pass the value of OUTPUT_ENABLED to call instead of its name, but let's go this way:
$ cat myfunction.mk
define call_script
set +x
mkdir -p $$(dirname $(2))
if [ ! -f $(2) ]; then
echo "" > $(2)
fi
if [ "$($(4))" = "yes" ]; then
REDIRECT='| tee -a'
else
REDIRECT='&>>'
fi
echo '>> $(1)'
eval "$(1) $$REDIRECT $(2)"
RET_CODE=$$?
echo "exit_code is: $$RET_CODE"
if [ ! $$RET_CODE = 0 ]; then
echo "$(3) terminated with error $$RET_CODE"
exit $$RET_CODE
else
if [ ! -z "$(strip $(3))" ]; then
echo "$(3) done"
fi
fi
endef
$ cat Makefile
.ONESHELL:
include myfunction.mk
OUTPUT_ENABLED ?= yes
target:
$(call call_script,echo "test",reports/mylog.log,doing test,OUTPUT_ENABLED)
Note that I moved the .ONESHELL: in the main Makefile because it is probably better to not hide it inside an included file. Up to you.
The most problematic issue here is that if you pipe your commands, the exit code is the exit code of the last command in a pipe, e.g false | tee foo.log will exit with 0 as tee will most probably succeed. Note also that pipe only redirects stdout, so your log will not contain any stderr messages unless explicitly redirected.
Considering that piping commands influence exit code and lack of portability of $PIPESTATUS (most specifically not being supported in dash), I would try to avoid piping commands and use a temporary file for gathering output, i.e.:
$ cat Makefile
# $(1) - script to execute
# $(2) - log file
# $(3) - description
define call_script
echo '>> $(1)'
$(if $(OUTPUT_ENABLED), \
$(1) > $#.log 2>&1; RET_CODE=$$?; mkdir -p $(dir $(2)); cat $#.log >> $(2); cat $#.log; rm -f $#.log, \
$(1); RET_CODE=$$? \
); \
echo "EXIT_CODE is: $${RET_CODE}"; \
if [ $${RET_CODE} -ne 0 ]; then $(if $(3),echo "$(3) terminated with error $${RET_CODE}";) exit $${RET_CODE}; fi; \
$(if $(3), echo "$(3) done.")
endef
good:
$(call call_script,echo "test",reports/mylog.log,doing test)
bad:
$(call call_script,mkdir /root/foo,reports/mylog.log,intentional fail)
ugly:
$(call call_script,bad_command,reports/mylog.log)
Regular call will not create the logs and will stop on errors:
$ make good bad ugly
echo '>> echo "test"'
>> echo "test"
echo "test"; RET_CODE=$? ; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "doing test terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi; echo "doing test done."
test
EXIT_CODE is: 0
doing test done.
echo '>> mkdir /root/foo'
>> mkdir /root/foo
mkdir /root/foo; RET_CODE=$? ; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "intentional fail terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi; echo "intentional fail done."
mkdir: cannot create directory ‘/root/foo’: Permission denied
EXIT_CODE is: 1
intentional fail terminated with error 1
make: *** [Makefile:19: bad] Error 1
Note that ugly was not built due to failure on bad. Now the same with the log:
$ make good bad ugly OUTPUT_ENABLED=1
echo '>> echo "test"'
>> echo "test"
echo "test" > good.log 2>&1; RET_CODE=$?; mkdir -p reports/; cat good.log >> reports/mylog.log; cat good.log; rm -f good.log; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "doing test terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi; echo "doing test done."
test
EXIT_CODE is: 0
doing test done.
echo '>> mkdir /root/foo'
>> mkdir /root/foo
mkdir /root/foo > bad.log 2>&1; RET_CODE=$?; mkdir -p reports/; cat bad.log >> reports/mylog.log; cat bad.log; rm -f bad.log; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "intentional fail terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi; echo "intentional fail done."
mkdir: cannot create directory ‘/root/foo’: Permission denied
EXIT_CODE is: 1
intentional fail terminated with error 1
make: *** [Makefile:19: bad] Error 1
$ cat reports/mylog.log
test
mkdir: cannot create directory ‘/root/foo’: Permission denied
Note that this time ugly was also not run. But if run later, it will correctly append to the log:
$ make ugly OUTPUT_ENABLED=1
echo '>> bad_command'
>> bad_command
bad_command > ugly.log 2>&1; RET_CODE=$?; mkdir -p reports/; cat ugly.log >> reports/mylog.log; cat ugly.log; rm -f ugly.log; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then exit ${RET_CODE}; fi;
/bin/sh: 1: bad_command: not found
EXIT_CODE is: 127
make: *** [Makefile:22: ugly] Error 127
$ cat reports/mylog.log
test
mkdir: cannot create directory ‘/root/foo’: Permission denied
/bin/sh: 1: bad_command: not found
Personally I am not fan of implementing logging in this way. It is complicated and it only logs output of commands, not make output itself, and only of those commands which are explicitly called to do so. I'd rather keep Makefile clean and simple and just run make 2>&1 | tee log instead to have the output logged.
If there is an alternative to bash, I will appreciate it too.
I have a large dump of MySQL commands (over 10 GB)
When restoring the dump I get a few warnings and occasionally an error. I need to execute those commands and process all warnings and errors. And, preferibly to do it automatically.
mysql --show-warnings
tee logfile.log
source dump.sql
The logfile will contain many lines telling each command was successful, and will display some warnings, particulartly truncate colums. But the original file has tens of thousands of very large INSERTs, the log is not particularly helpful. Despite it requires some kind of supervised interaction. (I cannot program a crontab, for example.)
#!/bin/bash
echo "tee logfile.log" > script.sql
echo "source $1" > script.sql
mysql --show-warnings < script.sql > tmpfile.log 2>&1
cat tmpfile.log >> logfile.log
The tee command doesn't work in this batch environment. I can capture all the warnings, but I cannot figure out which command produced each warning.
So I came down with this small monstruosity:
#!/bin/bash
ERRFILE=$(basename "$0" .sh).err.log
LOGFILE=$(basename "$1" .sql).log
log_action() {
WARN=$(cat)
[ -z "$WARN" ] || echo -e "Line ${1}: ${WARN}\n${2}" >> "$LOGFILE"
}
echo 0 > "$ERRFILE"
log_error() {
ERNO=$(cat "$ERRFILE")
ERR=$(cat)
[ -z "$ERR" ] || echo -e "*** ERROR ***\nLine ${1}: ${ERR}\n${2}" >> "$LOGFILE"
(( ERNO++ ))
echo $ERNO > "$ERRFILE"
}
COUNT=0
COMMAND=''
echo -e "**** BEGIN $(date +%Y-%m-%d\ %H:%M:%S)\n" > "$LOGFILE"
exec 4> >(log_action $COUNT "$COMMAND")
exec 5> >(log_error $COUNT "$COMMAND")
exec 3> >(mysql --show-warnings >&4 2>&5)
while IFS='' read -r LINE || [[ -n "$line" ]]
do
(( COUNT++ ))
[ ${#LINE} -eq 0 ] && continue # discard blank lines
[ "${LINE:0:2}" = "--" ] && continue # discard comments
COMMAND+="$LINE" # build command
[ "${LINE: -1}" != ";" ] && continue # if not finnished keep building
echo $COMMAND >&3 # otherwise execute
COMMAND=''
done < "$1"
exec 3>$-
exec 5>$-
exec 4>$-
echo -e "**** END $(date +%Y-%m-%d\ %H:%M:%S)\n" >> "$LOGFILE"
ERRS=$(cat "$ERRFILE")
[ "ERRS" = 0 ] || echo "${ERRS} Errors." >&2
This scans the file at $1 and sends the commands to an open MySQL connection at &3. That part is working fine.
The capture of warnings and errors is not working though.
It only records the first error.
It only records the first warning.
I haven't find a good way to pass the line number $COUNT and offending command $COMMAND to the recording functions.
The only error is after the time stamps, and the only warning is after the error, which is not the chronology of the script.
i have got some problems in my expect script.
I am spawning a scp file upload but the eof is not recognized.
The expect rans into the timeout before the script goes on.
What am i doing wrong??
Here is the script
function ssh_upload_file () {
test_log INFO "Uploading File $3 to Device $2 with a $1 seconds timeout" "ssh_upload_file"
last_log_entry_to_detail
expect <<- DONE
set outfilename "$DETAIL_LOG"
log_file "$DETAIL_LOG";
set timeout $1
set lantime_ip $2
system "touch ~/.ssh/known_hosts"
system "ssh-keygen -R $2"
spawn /bin/bash
sleep 3
send_user "Uploading File via SSH...\n"
send "scp $2 $3:$4 || echo FWTEST_SSH_RESULT=\$?\n"
expect "assword:" {
after 1000
send "$DUT_PWD\r"
#set timeout 5
#expect "y/n"
#send "n\n"
#set timeout $1
#expect "#"
#send "no_shell_timeout\n"
}
expect eof
send_user "Done."
log_file
exit 0
expect eof
DONE
LAST_ERROR=$?
return $LAST_ERROR
}
You spawn a bash shell, and send an scp command. When scp completes, you're sitting at a bash prompt. You need to send bash an exit command before you'll see eof.
Alternatively, you're not doing anything in that bash session except capture the output of the scp command which you can do like this:
spawn scp $2 $3:$4
expect "assword:"
send "$DUT_PWD\r"
expect eof
set info [wait]
puts "FWTEST_SSH_RESULT=[lindex [set info] 3]"
Addressing Donal's comment:
spawn scp $2 $3:$4
expect {
"assword:" {
send "$DUT_PWD\r"
exp_continue
}
eof
}
set info [wait]
puts "FWTEST_SSH_RESULT=[lindex [set info] 3]"
i have return a command to invoke my RTL compiler (Cadence Tool)using tcl script line
puts [exec rc -f script.g ]
i am getting error like abort child process. while if write it directly on my console the to invoke the tool $-rc -f script.g it is getting perfectly exectuded.
exec returns an error condition if:
the exec'ed command returns with a non-zero status
or it prints to stderr
Do this
if {[catch {exec rc -f script.g} output] == 0} {
# normal exit status
puts $output
} elseif {$errorCode eq NONE} {
# normal exit status, but some output to stderr (contained in $output)
puts $output
} else {
# some error condition.
# see http://wiki.tcl.tk/1039 for ways to handle
puts "some error occurred: $errorCode"
}
I am trying to create an expect script which will grep a file and return the line which contains the string I am looking for, in this case the string will be a terminal ID. For example I have a file called terminal_list.txt with the following contents:
0x400 192.168.62.133 10006
0x420 192.168.62.133 10021
0x440 192.168.62.133 10022
and I want the line returned which starts with 0x420
My code is as follows:
#!/usr/bin/expect -f
set terminal_list "/home/linkway/terminal_list.txt"
set termid "0x400"
spawn /bin/bash
expect "] "
# Get ip and port of terminal
# Check if termid exists in terminal_list file
set command "grep -q '$termid' '$terminal_list' && echo 'true' || echo 'false'"
send "$command\r"
expect "$command\r\n"
expect -re "(.*)\r\n.*] "
set val $expect_out(1,string)
# If terminal_list does not exist print error and exit
if { [string first "No such file or directory" $val] >= 0 } {
puts "$terminal_list does not exist. Script Failed"
exit
# If terminal_list does not contain termid print error and continue
} elseif { [string match "false" $val] } {
puts "\nTerminal $termid does not exist in ${terminal_list}. Cannot update bootfile.\n"
# If termid is found in terminal_list, get the ip and port of the terminal
} else {
set command "grep '$termid' '$terminal_list'"
send "$command\r"
expect "$command\r\n"
expect -re "(.*)\r\n.*] "
set val $expect_out(1,string)
set ip_port [string range $val 6 end]
}
This works perfectly when I ssh to the RHEL server via putty and run the script in a maximized putty window. HOWEVER when I shrink the window length so that the grep command no longer fits on a single line my code breaks! Can anyone please help me come up with a solution to this? I have been struggling with the processing of expect_out and could really use some guidance.
EDIT: I found what was causing the bug. It turns out that when the grep command is split over multiple lines, a \r is added to the command where the line break is. Here is some debugging info from exp_internal 1. You can see how the \r is added into the grep command where the command ran onto the next line:
expect: does "grep -q '0x400' '/home/linkway/term \rinal_list.txt'
&& echo 'true' || echo 'false'\r\n" (spawn_id exp6)
match glob pattern "grep -q '0x400' '/home/linkway/terminal_list.txt'
&& echo 'true' || echo 'false'\r\n"? no
Why is this happening and what is the proper way to get just the output of the grep command? I find it very strange that expect would behave differently base on how the output of a command is displayed on screen. Any help is greatly appreciated.
I was able to find a cleaner solution to my problem by making my script more expect-like. Here's what it looks like:
set command "grep -q '$termid' '$terminal_list' && echo 'true' || echo 'false'"
send "$command\r"
expect {
-re ".*\r\ntrue\r\n.*] " {
send "grep '$termid' '$terminal_list'\r"
expect ".*\r\n"
expect -re "(.*)\r\n.*] "
set val $expect_out(1,string)
set ip_port [string range $val 6 end]
puts "\nUpdating $termid bootfile"
updatebootfile $ip_port $boot_data $termid
}
-re ".*\r\nfalse\r\n.*] " {
puts "\nTerminal $termid does not exist in ${terminal_list}. Cannot update bootfile.\n"
}
-re "No such file or directory.*] " {
puts "$terminal_list does not exist. Script Failed"
exit 1
}
timeout {
puts "expect timeout when searching for $termid in $terminal_list"
}
}