I have a makefile function inside a makefile (myfunction.mk):
.ONESHELL:
define call_script
set +x
mkdir -p $$(dirname $(2))
if [ ! -f $(2) ]; then
echo "" > $(2)
fi
REDIRECT='| tee -a'
echo '>> $(1)'
($(1) ???????? $(2))
RET_CODE=$$?
echo "exit_code is: $$RET_CODE"
if [ ! $$RET_CODE = 0 ]; then
echo "$(3) terminated with error $$RET_CODE"
exit $$RET_CODE
else
if [ ! -z "$(strip $(3))" ]; then
echo "$(3) done"
fi
fi
endef
this function call a script and append result to a log (which is created with its folder if non existing), the result of the script is append only if the makefile variable given as the 4th ($(4)) argument is equal to 'yes'.
you call it like this:
include myfunction.mk
OUTPUT_ENABLED ?= yes
target:
$(call call_script, echo "test", reports/mylog.log, "doing test", OUTPUT_ENABLED)
This works for the most part:
if i replace '????????' by '| tee -a', it works.
if i replace '????????' by $(REDIRECT), it fails.
if i replace '????????' by $$REDIRECT, it fails.
why?
note: running it from a shell /bin/sh: symbolic link to dash
note: of course i want to add a ifeq that allows me to check for $(4) and replace | tee -a by &>>
I'll assume that you use call in a recipe, not flat in your Makefile. There are few problems with your shell script. First, if you try the following on the command line:
mkdir -p reports
REDIRECT='| tee -a'
echo '>> echo "test"'
(echo "test" $REDIRECT reports/mylog.log)
you'll see that echo considers:
"test" $REDIRECT reports/mylog.log
as its arguments. They are expanded and echoed, which prints:
test | tee -a reports/mylog.log
on the standard output, not the effect you expected, I guess. You could, for instance, use eval. On the command line:
eval "echo "test" $REDIRECT reports/mylog.log"
Which, in your Makefile, would become:
eval "$(1) $$REDIRECT $(2)"
Next you should not quote the third parameter of call because the quotes will be passed unmodified and your script will be expanded by make as:
echo " "doing test" terminated with error $RET_CODE"
Again probably not what you want.
Third, you should avoid useless spaces in the parameters of call because they are preserved too (as you can see above between the first 2 double quotes):
.PHONY: foo
foo:
$(call call_script,echo "test",reports/mylog.log,doing test,OUTPUT_ENABLED)
And for your last desired feature, it would be slightly easier to pass the value of OUTPUT_ENABLED to call instead of its name, but let's go this way:
$ cat myfunction.mk
define call_script
set +x
mkdir -p $$(dirname $(2))
if [ ! -f $(2) ]; then
echo "" > $(2)
fi
if [ "$($(4))" = "yes" ]; then
REDIRECT='| tee -a'
else
REDIRECT='&>>'
fi
echo '>> $(1)'
eval "$(1) $$REDIRECT $(2)"
RET_CODE=$$?
echo "exit_code is: $$RET_CODE"
if [ ! $$RET_CODE = 0 ]; then
echo "$(3) terminated with error $$RET_CODE"
exit $$RET_CODE
else
if [ ! -z "$(strip $(3))" ]; then
echo "$(3) done"
fi
fi
endef
$ cat Makefile
.ONESHELL:
include myfunction.mk
OUTPUT_ENABLED ?= yes
target:
$(call call_script,echo "test",reports/mylog.log,doing test,OUTPUT_ENABLED)
Note that I moved the .ONESHELL: in the main Makefile because it is probably better to not hide it inside an included file. Up to you.
The most problematic issue here is that if you pipe your commands, the exit code is the exit code of the last command in a pipe, e.g false | tee foo.log will exit with 0 as tee will most probably succeed. Note also that pipe only redirects stdout, so your log will not contain any stderr messages unless explicitly redirected.
Considering that piping commands influence exit code and lack of portability of $PIPESTATUS (most specifically not being supported in dash), I would try to avoid piping commands and use a temporary file for gathering output, i.e.:
$ cat Makefile
# $(1) - script to execute
# $(2) - log file
# $(3) - description
define call_script
echo '>> $(1)'
$(if $(OUTPUT_ENABLED), \
$(1) > $#.log 2>&1; RET_CODE=$$?; mkdir -p $(dir $(2)); cat $#.log >> $(2); cat $#.log; rm -f $#.log, \
$(1); RET_CODE=$$? \
); \
echo "EXIT_CODE is: $${RET_CODE}"; \
if [ $${RET_CODE} -ne 0 ]; then $(if $(3),echo "$(3) terminated with error $${RET_CODE}";) exit $${RET_CODE}; fi; \
$(if $(3), echo "$(3) done.")
endef
good:
$(call call_script,echo "test",reports/mylog.log,doing test)
bad:
$(call call_script,mkdir /root/foo,reports/mylog.log,intentional fail)
ugly:
$(call call_script,bad_command,reports/mylog.log)
Regular call will not create the logs and will stop on errors:
$ make good bad ugly
echo '>> echo "test"'
>> echo "test"
echo "test"; RET_CODE=$? ; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "doing test terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi; echo "doing test done."
test
EXIT_CODE is: 0
doing test done.
echo '>> mkdir /root/foo'
>> mkdir /root/foo
mkdir /root/foo; RET_CODE=$? ; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "intentional fail terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi; echo "intentional fail done."
mkdir: cannot create directory ‘/root/foo’: Permission denied
EXIT_CODE is: 1
intentional fail terminated with error 1
make: *** [Makefile:19: bad] Error 1
Note that ugly was not built due to failure on bad. Now the same with the log:
$ make good bad ugly OUTPUT_ENABLED=1
echo '>> echo "test"'
>> echo "test"
echo "test" > good.log 2>&1; RET_CODE=$?; mkdir -p reports/; cat good.log >> reports/mylog.log; cat good.log; rm -f good.log; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "doing test terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi; echo "doing test done."
test
EXIT_CODE is: 0
doing test done.
echo '>> mkdir /root/foo'
>> mkdir /root/foo
mkdir /root/foo > bad.log 2>&1; RET_CODE=$?; mkdir -p reports/; cat bad.log >> reports/mylog.log; cat bad.log; rm -f bad.log; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then echo "intentional fail terminated with error ${RET_CODE}"; exit ${RET_CODE}; fi; echo "intentional fail done."
mkdir: cannot create directory ‘/root/foo’: Permission denied
EXIT_CODE is: 1
intentional fail terminated with error 1
make: *** [Makefile:19: bad] Error 1
$ cat reports/mylog.log
test
mkdir: cannot create directory ‘/root/foo’: Permission denied
Note that this time ugly was also not run. But if run later, it will correctly append to the log:
$ make ugly OUTPUT_ENABLED=1
echo '>> bad_command'
>> bad_command
bad_command > ugly.log 2>&1; RET_CODE=$?; mkdir -p reports/; cat ugly.log >> reports/mylog.log; cat ugly.log; rm -f ugly.log; echo "EXIT_CODE is: ${RET_CODE}"; if [ ${RET_CODE} -ne 0 ]; then exit ${RET_CODE}; fi;
/bin/sh: 1: bad_command: not found
EXIT_CODE is: 127
make: *** [Makefile:22: ugly] Error 127
$ cat reports/mylog.log
test
mkdir: cannot create directory ‘/root/foo’: Permission denied
/bin/sh: 1: bad_command: not found
Personally I am not fan of implementing logging in this way. It is complicated and it only logs output of commands, not make output itself, and only of those commands which are explicitly called to do so. I'd rather keep Makefile clean and simple and just run make 2>&1 | tee log instead to have the output logged.
I'm writing a shell code to automaticaly run multiple sql queries in multiple databases. My code is working well, but beside "select" queries, all other queries aren't displaying anything in the prompt while executing. How could I force query outputs to redirect to prompt?
That's some of my code:
for x in "${db[#]}"
do
found=0
for enreg in `cat /home/dbfile.csv`
do
#extracting database data from csv
DBNAME=`echo $enreg | awk -F";" '{ print $4 }'`
if [ $x = $DBNAME ]
then
PASS=`echo $enreg | awk -F";" '{ print $2 }'`
HOST=`echo $enreg | awk -F";" '{ print $3 }'`
USERNAME=`echo $enreg | awk -F";" '{ print $1 }'`
# Running queries in database $DBNAME
for y in "${req[#]}"
do
echo "
"
mysql -u $USERNAME -p$PASS -h $HOST $DBNAME -e "$y"
echo "
"
done
found=1
break
fi
done
if [ $found -eq 0 ]
then
echo "Database $x doesn't exist"
fi
done
I writing a script, which run backup with rsync and the output of rsync insert to database.
Its working well except one little thing. The text in the DB is one line and there is not 'new line' bunch of spaces or tabulator. However if I writing the query to the screen, is fine. Its looking what i want in db cell.
Can anyone help in this issue?
Here is the script:
#!/bin/bash
DATUM=$(date +%Y-%m-%d)
IDO=$(date +%H:%M:%S)
LOG_FAJL="/media/2TB/MENTES_PC/log/"$DATUM"_"$IDO"_rsync_2TB.log"
LOG_FAJL_SED="/media/2TB/MENTES_PC/log/"$DATUM"_"$IDO"_rsync_2TB_SED.log"
CEL="/media/BACKUP_2TB/"
FORRAS="/media/2TB/"
KIZARAS1="--exclude='/media/2TB/lost+found/'" -- exclude='/media/2TB/.Trash-1000/'
DB_USER="user"
DB_PASSWORD="secret"
DB="xx"
TABLA="table"
if [ ! -f "$LOG_FAJL" ];
then
printf "%b" "\\\\n\\n\\t\\t\\t\\t\\t\\t\\t\\t##########################################" > "$LOG_FAJL" # more fancy header...
fi
printf "%b" "\\n\\n\\n A mentés készült: $(date +%Y-%m-%d_%H:%M:%S) \\n" >> "$LOG_FAJL"
printf "%b" " *************************************\\n" >> "$LOG_FAJL"
KEZDES=$(date +%s)
printf "%b" "\\n "$FORRAS" mentése ide: "$CEL >> "$LOG_FAJL"
printf "%b" "\\n ==========================================\\n\\n" >> "$LOG_FAJL"
rsync -azvh --delete "$KIZARAS1" --stats --human-readable $FORRAS $CEL >> "$LOG_FAJL"
VEGE=$(date +%s)
ELTELT=$(($VEGE-$KEZDES))
printf "%b" "\\n\\n A mentés tartott: " >> "$LOG_FAJL"
printf "%02dh:%02dm:%02ds" "$(($ELTELT/3600))" "$(($ELTELT%3600/60))" "$(($ELTELT%60))" >> "$LOG_FAJL"
printf "%b" "-ig\\n ********************************" >> "$LOG_FAJL"
cat "$LOG_FAJL" | sed "s/'/\\\'/g" > "$LOG_FAJL_SED"
QUERY="INSERT INTO "$TABLA" (datum,ido,log) VALUES ('"$DATUM"','"$IDO"','"$(cat "$LOG_FAJL_SED")"');"
echo "$QUERY"
mysql -u$DB_USER -p$DB_PASSWORD $DB -e "$QUERY"
Curious what application you are using to view the inserted data, Mysql Workbench, Squirrel, etc?
I suspect that the newline/carriage return is actually there, it is just that the app is not showing them.
You can use mysqldump to extract the data to a file and see if the newlines are in the output.
If there is an alternative to bash, I will appreciate it too.
I have a large dump of MySQL commands (over 10 GB)
When restoring the dump I get a few warnings and occasionally an error. I need to execute those commands and process all warnings and errors. And, preferibly to do it automatically.
mysql --show-warnings
tee logfile.log
source dump.sql
The logfile will contain many lines telling each command was successful, and will display some warnings, particulartly truncate colums. But the original file has tens of thousands of very large INSERTs, the log is not particularly helpful. Despite it requires some kind of supervised interaction. (I cannot program a crontab, for example.)
#!/bin/bash
echo "tee logfile.log" > script.sql
echo "source $1" > script.sql
mysql --show-warnings < script.sql > tmpfile.log 2>&1
cat tmpfile.log >> logfile.log
The tee command doesn't work in this batch environment. I can capture all the warnings, but I cannot figure out which command produced each warning.
So I came down with this small monstruosity:
#!/bin/bash
ERRFILE=$(basename "$0" .sh).err.log
LOGFILE=$(basename "$1" .sql).log
log_action() {
WARN=$(cat)
[ -z "$WARN" ] || echo -e "Line ${1}: ${WARN}\n${2}" >> "$LOGFILE"
}
echo 0 > "$ERRFILE"
log_error() {
ERNO=$(cat "$ERRFILE")
ERR=$(cat)
[ -z "$ERR" ] || echo -e "*** ERROR ***\nLine ${1}: ${ERR}\n${2}" >> "$LOGFILE"
(( ERNO++ ))
echo $ERNO > "$ERRFILE"
}
COUNT=0
COMMAND=''
echo -e "**** BEGIN $(date +%Y-%m-%d\ %H:%M:%S)\n" > "$LOGFILE"
exec 4> >(log_action $COUNT "$COMMAND")
exec 5> >(log_error $COUNT "$COMMAND")
exec 3> >(mysql --show-warnings >&4 2>&5)
while IFS='' read -r LINE || [[ -n "$line" ]]
do
(( COUNT++ ))
[ ${#LINE} -eq 0 ] && continue # discard blank lines
[ "${LINE:0:2}" = "--" ] && continue # discard comments
COMMAND+="$LINE" # build command
[ "${LINE: -1}" != ";" ] && continue # if not finnished keep building
echo $COMMAND >&3 # otherwise execute
COMMAND=''
done < "$1"
exec 3>$-
exec 5>$-
exec 4>$-
echo -e "**** END $(date +%Y-%m-%d\ %H:%M:%S)\n" >> "$LOGFILE"
ERRS=$(cat "$ERRFILE")
[ "ERRS" = 0 ] || echo "${ERRS} Errors." >&2
This scans the file at $1 and sends the commands to an open MySQL connection at &3. That part is working fine.
The capture of warnings and errors is not working though.
It only records the first error.
It only records the first warning.
I haven't find a good way to pass the line number $COUNT and offending command $COMMAND to the recording functions.
The only error is after the time stamps, and the only warning is after the error, which is not the chronology of the script.
I'm in the middle of making my first django app, and I'd like to commit it to git in such a way that someone can clone it down and start working on it with the least amount of trouble. One of the things I needed to do to get things up and running was to create a new db in my local mysql installation and create a new user there. I'd love to let someone clone things down and have that done automatically for them. Is there a good way to do this?
Use mysql-python and write a python script to create the database or alter my shell script to make it suit your needs.
#!/bin/bash
function pre_checks() {
if [[ "$1" -ne 3 ]]; then
echo "Usage: $0 [DATABASE NAME] [USERNAME] [HOST]"
return 1
fi
if ! command -v /usr/bin/mysql >/dev/null 2>&1; then
echo "Mysql is not installed."
return 1
fi
echo -n "Create the database '${2}' and the user '${3}' now? (y/n) "
read ANSWER
case "$ANSWER" in
"y"|"Y")
echo -n "Password for ${3}: "
read -s USER_PW
echo
return 0 ;;
"n"|"N"| *)
echo "Bye."
return 1 ;;
esac
}
function create_db() {
Q1="CREATE DATABASE IF NOT EXISTS ${1} CHARACTER SET utf8;"
Q2="GRANT ALL ON *.* TO '${2}'#'${3}' IDENTIFIED BY '$USER_PW';"
Q3="FLUSH PRIVILEGES;"
Q4="SHOW DATABASES;"
SQL="${Q1} ${Q2} ${Q3} ${Q4}"
echo "Query:"
echo "${SQL}"
echo -n "Run query now? (y/n) "
read ANSWER
case "$ANSWER" in
"y" | "Y" )
/usr/bin/mysql -uroot -p -e "$SQL" || echo "Failure."
;;
"n" | "N" | *)
echo "Bye."
return 1
;;
esac
}
pre_checks "$#" "$1" "$2" && create_db "$1" "$2" "$3"