Shell script: variable scope in functions - function

I wrote a quick shell script to emulate the situation of xkcd #981 (without hard links, just symlinks to parent dirs) and used a recursive function to create all the directories. Unfortunately this script does not provide the desired result, so I think my understanding of the scope of variable $count is wrong.
How can I properly make the function use recursion to create twenty levels of folders, each containing 3 folders (3^20 folders, ending in soft links back to the top)?
#!/bin/bash
echo "Generating folders:"
toplevel=$PWD
count=1
GEN_DIRS() {
for i in 1 2 3
do
dirname=$RANDOM
mkdir $dirname
cd $dirname
count=$(expr $count + 1)
if [ $count < 20 ] ; then
GEN_DIRS
else
ln -s $toplevel "./$dirname"
fi
done
}
GEN_DIRS
exit

Try this (amended version of the script) — it seems to work for me. I decline to test to 20 levels deep, though; at 8 levels deep, each of the three top-level directories occupies some 50 MB on a Mac file system.
#!/bin/bash
echo "Generating folders:"
toplevel=$PWD
GEN_DIRS()
{
cur=${1:?}
max=${2:?}
for i in 1 2 3
do
dirname=$RANDOM
if [ $cur -le $max ]
then
(
echo "Directory: $PWD/$dirname"
mkdir $dirname
cd $dirname
GEN_DIRS $((cur+1)) $max
)
else
echo "Symlink: $PWD/$dirname"
ln -s $toplevel "./$dirname"
fi
done
}
GEN_DIRS 1 ${1:-4}
Lines 6 and 7 are giving names to the positional parameters ($1 and $2) passed to the function — the ${1:?} notation simply means that if you omit to pass a parameter $1, you get an error message from the shell (or sub-shell) and it exits.
The parentheses on their own (lines 13 and 18 above) mean that the commands in between are run in a sub-shell, so changes in directory inside the sub-shell do not affect the parent shell.
The condition on line 11 now uses arithmetic (-le) instead of string < comparisons; this works better for deep nesting (because the < is a lexicographic comparison, so level 9 is not less than level 10). It also means that the [ command is OK to use instead of the [[ command (although [[ would also work, I prefer the old-fashioned notation).

I end up creating a script like this:
#!/bin/bash
echo "Generating folders:"
toplevel=$PWD
level=0
maxlevel=4
function generate_dirs {
pushd "$1" >/dev/null || return
(( ++level ))
for i in 1 2 3; do
dirname=$RANDOM
if (( level < maxlevel )); then
echo "$PWD/$dirname"
mkdir "$dirname" && generate_dirs "$dirname"
else
echo "$PWD/$dirname (link to top)"
ln -sf "$toplevel" "$dirname"
fi
done
popd >/dev/null
(( --level ))
}
generate_dirs .
exit

Related

Newbie: unix bash, nested if statement, results from a loop results from sql

Newbie here, please pardon any confusing wording that I use.
A common task I have is to take a list of names and do a MySQL query to look the names up in a table and see if they are "live" on our site.
Doing this one at a time, my SQL query works fine. I then wanted to do the query using a loop from a file listing multiple names. This works fine, too.
I added this query loop to my bash profile so that I can quickly do the task by typing this:
$ ValidOnSite fileName
This works fine, and I even added an usage statement for my process to remind myself of the syntax. Below is what I have that works fine:
validOnSite() {
if [[ "$1" == "" ]] || [[ "$1" == "-h" ]] || [[ "$1" == "--help" ]]; then
echo "Usage:"
echo " $ validOnSite [filename]"
echo " Where validOnSite uses specified file as variables in sql query:"
echo " SELECT name, active FROM dbDb WHERE name=lines in file"
else
cat $1 | while read line ; do hgsql -h genome-centdb hgcentral -Ne "select name, active from dbDb where name='$line'" ; done
fi
Using a file "list.txt" which contains:
nameA
nameB
I would then type:
validOnSite list.txt
and both entries in list.txt meet my query criteria and are found in sql. My results will be:
nameA 1
nameB 1
Note the "1" after each result. I assume this is some sort of "yes" status.
Now, I add a third name to my list.txt, one that I know is not a match in sql. Now list.txt contains:
nameA
nameB
foo
When I again run this command for my list with 3 rows:
validOnSite list.txt
My results are the same as when I used the 1st version of file.txt, and I cannot see which lines failed, I still only see which lines were a success:
nameA 1
nameB 1
I have been trying all kinds of things to add a nested if statement, something that says, "If $line is a match, echo "pass", else echo "fail."
I do not want to see a "1" in my results. Using file.txt with 2 matches and 1 non-match, I would like my results to be:
nameA pass
nameB pass
foo fail
Or even better, color code a pass with green and a fail with red.
As I said, newbie here... :)
Any pointers in the right direction would help. Here is my latest sad attempt, but I realize I may be going in a wrong direction entirely:
validOnSite() {
if [[ "$1" == "" ]] || [[ "$1" == "-h" ]] || [[ "$1" == "--help" ]]; then
echo "Usage:"
echo " $ validOnSite [filename]"
echo " Where validOnSite uses specified file as variables in sql query:"
echo " SELECT name, active FROM dbDb WHERE name=lines in file"
else
cat $1 | while read line ; do hgsql -h genome-centdb hgcentral -Ne "select name, active from dbDb where name='$line'" > /dev/null ; done
if ( "status") then
echo $line "failed"
echo $line "failed" >> outfile
else
echo $line "ok"
echo $line "ok" >>outfile
clear
cat outfile
fi
fi
If something looks crazy in my last attempt, it's because it is - I am just googling around and trying as many things as I can while trying to learn. Any help appreciated, I feel stuck after working on this for a long time, but I am excited to move forward and find a solution! I think there is something I'm missing about understanding stdout, and also confusion about nested if's.
Note: I do not need an outfile, but it's ok if one is needed to accomplish the goal. stdout result alone would suffice, and is preferred.
Note: hgssql is just the name of our MySQL server. The MySQL part works fine, I am looking for a better way to deal with my bash output, and I think there is something about stderr that I'm missing. I'm looking for a fairly simple answer as I'm a newbie!
I guess, by hgsql you mean some Mercurial extension that allows to perform MySQL queries. I don't know how hgsql works, but I know that MySQL returns only the matching rows. But in terms of shell scripting, the result is a string that may contain extra information even if the number of matched rows is zero. For example, some MySQL client may return the header or a string like "No rows found", although it is unlikely.
I'll show how it is done with the official mysql client. I'm sure you will manage to adapt hgsql with the help of its documentation to the following example.
if [ -t 1 ]; then
red_color=$(tput setaf 1)
green_color=$(tput setaf 2)
reset_color=$(tput sgr0)
else
red_color=
green_color=
reset_color=
fi
colorize_flag() {
local color
if [ "$1" = 'fail' ]; then
color="$red_color"
else
color="$green_color"
fi
printf '%s' "${color}${1}${reset_color}"
}
sql_fmt='SELECT IF(active, "pass", "fail") AS flag FROM dbDb WHERE name = "%s"'
while IFS= read -r line; do
sql=$(printf "$sql_fmt" "$line")
flag=$(mysql --skip-column-names dbname -e "$sql")
[ -z "$flag" ] && flag='fail'
printf '%-20s%s\n' "$line" "$(colorize_flag "$flag")"
done < file
The first block detects if the script is running in interactive mode by checking if the file descriptor 1 (standard output) is opened on a terminal (see help test). If it is opened in a terminal, the script considers that the script is running interactively, i.e. the standard output is connected to the user's terminal directly, but not via pipe, for example. For interactive mode, it assigns variables to the terminal color codes with the help of tput command.
colorize_flag function accepts a string ($1) and outputs the string with the color codes applied according to its value.
The last block reads file line by line. For each line builds an SQL query string (sql) and invokes mysql command with the column names stripped off the output. The output of the mysql command is assigned to flag by means of command substitution. If "$flag" is empty, it is assigned to 'fail'. The $line and the colorized flag are printed to standard output.
You can test the non-interactive mode by chaining the output via pipe, e.g.:
./script | tee -a
I must warn you that it is generally bad idea to pass the shell variables into SQL queries unless the values are properly escaped. And the popular shells do not provide any tools to escape MySQL strings. So consider running the queries in Perl, PHP, or any programming language that is capable of building and running the queries safely.
Also note that in terms of performance it is better to run a single query and then parse the result set in a loop instead of running multiple queries in a loop, with the exception of prepared statements.
I found a way to get to my solution by piecing together the few basic things that I know. Not elegant, but it works well enough for now. I created a file "[filename]Results" with the output:
nameA 1
nameB 1
I then cut out the "1"s and made a new file. I then did a comparison with "[fileName]results" to list.txt in order to see what lines exist in file.txt but do not exist in results.
Note: I have the following in my .zshrc file.
validOnSite() {
if [[ "$1" == "" ]] || [[ "$1" == "-h" ]] || [[ "$1" == "--help" ]]; then
echo "Usage:"
echo " $ validOnSite [filename]"
echo " Where validOnSite uses specified file as variables in sql query:"
echo " SELECT name, active FROM dbDb WHERE name=lines in file"
else
cat $1 | while read line ; do hgsql -h genome-centdb hgcentral -Ne "select name from dbDb where name='$line' and active='1'" >> $1"Pass"; done
autoload -U colors
colors
echo $fg_bold[magenta]Assemblies active on site${reset_color}
echo
cat $1"Pass"
echo
echo $fg_bold[red]Not active or not found on site${reset_color}
comm -23 $1 $1"Pass" 2> /dev/null
echo
echo
mv $1"Pass" ~cath/myFiles/validOnSiteResults
echo "Results file containing only active assemblies resides in ~cath/myFiles/validOnSiteResults"
fi
}
list.txt:
nameA
nameB
foo
My input:
validOnSite list.txt
My output:
Assemblies active on site (<--this font is magenta)
nameA
nameB
Not active or not found on site (<--this font is red)
foo
Results file containing only active assemblies resides in ~me/myFiles/validOnRRresults

Periodically reading output from async background scripts

Context: I'm making my own i3-Bar script to read output from other (asynchronous) scripts running in background, concatenate them and then echo them to i3-Bar itself.
The way I'm passing outputs is in plain files, and I guess (logically) the problem is that the files are sometimes read and written at the same time. The best way to reproduce this behavior is by suspending the computer and then waking it back up - I don't know the exact cause of this, I can only go on what I see from my debug log files.
Main Code: Added comments for clarity
#!/usr/bin/env bash
cd "${0%/*}";
trap "kill -- -$$" EXIT; #The bg. scripts are on a while [ 1 ] loop, have to kill them.
rm -r ../input/*;
mkdir ../input/; #Just in case.
for tFile in ./*; do
#Run all of the available scripts in the current directory in the background.
if [ $(basename $tFile) != "main.sh" ]; then ("$tFile" &); fi;
done;
echo -e '{ "version": 1 }\n['; #I3-Bar can use infinite array of JSON input.
while [ 1 ]; do
input=../input/*; #All of the scripts put their output in this folder as separate text files
input=$(sort -nr <(printf "%s\n" $input));
output="";
for tFile in $input; do
#Read and add all of the files to one output string.
if [ $tFile == "../input/*" ]; then break; fi;
output+="$(cat $tFile),";
done;
if [ "$output" == "" ]; then
echo -e "[{\"full_text\":\"ERR: No input files found\",\"color\":\"#ff0000\"}],\n";
else
echo -e "[${output::-1}],\n";
fi;
sleep 0.2s;
done;
Example Input Script:
#!/usr/bin/env bash
cd "${0%/*}";
while [ 1 ]; do
echo -e "{" \
"\"name\":\"clock\"," \
"\"separator_block_width\":12," \
"\"full_text\":\"$(date +"%H:%M:%S")\"}" > ../input/0_clock;
sleep 1;
done;
The Problem
The problem isn't the script itself, but the fact, that i3-Bar receives a malformed JSON input (-> parse error), and terminates - I'll show such log later.
Another problem is, that the background scripts should run asynchronously, because some need to update every 1 second nad some only every 1 minute, etc. So the use of a FIFO isn't really an option, unless I create some ugly inefficient hacky stuff.
I know there is a need for IPC here, but I have no idea how to effieciently do this.
Script output from randomly crashing - waking up error looks the same
[{ "separator_block_width":12, "color":"#BAF2F8", "full_text":"192.168.1.104 "},{ "separator_block_width":12, "color":"#BAF2F8", "full_text":"100%"}],
[{ "separator_block_width":12, "color":"#BAF2F8", "full_text":"192.168.1.104 "},,],
(Error is created by the second line)
As you see, the main script tries to read the file, doesn't get any output, but the comma is still there -> malformed JSON.
The immediate error is easy to fix: don't append an entry to output if the corresponding file is empty:
for tFile in $input; do
[[ $tFile != "../input/*" ]] &&
[[ -s $tFile ]] &&
output+="$(<$tFile),"
done
There is a potential race condition here, though. Just because a particular input file exists doesn't mean that the data is fully written to it yet. I would change your input scripts to look something like
#!/usr/bin/env bash
cd "${0%/*}";
while true; do
o=$(mktemp)
printf '{"name": "clock", "separator_block_width": 12, "full_text": %(%H:%M:%S)T}\n' > "$o"
mv "$o" ../input/0_clock
sleep 1
done
Also, ${output%,} is a safer way to trim a trailing comma when necessary.

Passing arguments as strings to a function in a shell script

Calling a function in another script to delete old files - need to pass $1 as string, and not the eval of that arg (filelist from directory)
Have tried:
- single and double quotes around echo $1 ("$1", '$1')
- single and double quotes around arg ("/tmp/AB*", '/tmp/AB*')
Have read 3 similar questions here, but unsuccessful at understanding the issue...
AIX 6
#!/bin/ksh
#### common load function ######
. /tmp/functions.sh
deletefiles /usr/tmp/AB* 1
#!/bin/sh
# Deletes files from a filelist that are older than X days
deletefiles() {
echo $1
echo $2
#filelist=$1
#days=$2
#execute
#`find ${filelist} -type f -mtime +${days} -exec rm {} + 2>&1`
}
It looks like you want to pass /usr/tmp/AB* as is, without expanding it. This can be done with '/usr/tmp/AB*', "/usr/tmp/AB*", or /usr/tmp/AB\*.
Then, to confirm that you got the right value, you need to use "$1" to prevent wildcard expansion in echo:
deletefiles() {
echo "$1"
echo "$2"
}
deletefiles '/usr/tmp/AB*' 1
I guess your main problem is that you want AB* expanded in deletefiles().
When you don't do something special, how do you find the last parameter?
You can expand the wildcard within deletefiles() with eval, but eval can do more than you wanted. Another method is swithing the order of your parameters (days first) end use shift for deleting days from the paramaterlist when you assigned it to a var.
I'll show both solutions.
deletefiles_notsecure() {
filelist="$(eval echo $1)"
days=$2
echo "Filelist: $filelist"
echo "Days: $days"
}
deletefiles_secure() {
days=$1
shift
filelist="$*"
echo "Filelist: $filelist"
echo "Days: $days"
}
# deletefiles /usr/tmp/AB* 1
deletefiles_notsecure "/tmp/*" 1
echo ===========
deletefiles_secure 1 /tmp/*
As you can see, the second form can be used without quotes from the caller, so that will be easier to use.
Note: It will expand the vars during the call, relative to path you are standing in. When deletefiles_secure() starts with cd "${logdir}" and you are standing in your $HOME when you call deletefiles_secure 1 access*.log* an access.log in your homedir will be found. Use full paths on your local computer!
Don't use eval if you can avoid it: find does have a -name option to specify a file-mask, eg:
deletefiles () {
find "$1" -name "$2" ...
}
deletefiles /somedir 'AB*'

unix function return if any error occurs

I have a unix script in which I am calling functions.
I want the function should return immediately if any of the command failed in between.
But checking $? after every command I can not do. Is there any other way to do this.
Maybe running the script from a file line by line (as long of course as each of your functions are one line long).
Maybe the following script can be a starting point:
#!/bin/sh
while read l
do
eval "$l || break"
done <<EOF
echo test | grep e
echo test2 | grep r
echo test3 grep 3
EOF
This is another idea after my previous answer. It works with bash script and requires your functions to be quite simple (pipes may cause some issues):
#!/bin/bash
set -o monitor
check() {
[ $? -eq 0 ] && exit
}
trap check SIGCHLD
/bin/echo $(( 1+1 ))
/bin/echo $(( 1/0 ))
/bin/echo $(( 2+2 ))
Furthermore: functions need to be external command (this is why I use /bin/echo rather than echo). Regards.

find and replace script (difficult issue...NEED HELP!)

I've written a function in zsh to find and replace a specific number with a keyword that I'll use later on in a larger script. Here's what I've got:
function replace_metal() {
for file in "$#"; do
[ -f "$file" ] && mv $file $file.old
# replace metal
awk '/^28\s/ { gsub(/28\s/, "METAL") }; { print }' $file.old > $file
# remove temporary files
rm -f $file.old
done
}
The awk portion works fine when I run it on the command line but while in the script, it fails to parse the file and replace the number with the keyword. I'm not sure why it fails. I've written a function that is similar that works without any trouble:
function fix_filename() {
for file in "$#"; do
[ -f "$file" ] && mv $file $file.old
# fix filename
awk '{ gsub(/myFileName/,FILENAME); print }' $file.old > $file.tmp
# clean up filename
awk '{ gsub(/.gjf.old/,""); print }' $file.tmp > $file
# remove temporary files
rm -f $file.old $file.tmp
done
}
I'm especially confused as to why awk won't work in the replace_metal function but will on the command line. If anyone can explain that, I'd really appreciate it.
Here's an example portion of a file that I'd run this script on. They are cartesian coordinates for a molecular geometry program I use.
6 4.387152 -0.132561 1.145384
6 4.435130 0.035315 -0.261758
6 3.241800 0.069735 -1.002575
7 2.023205 -0.053248 -0.382329
6 1.948032 -0.217668 0.977856
6 3.120408 -0.260395 1.759133
8 0.936529 -0.001059 -1.144164
28 -0.810634 -0.374713 -0.376819
7 -1.066408 1.593331 -0.221421
6 -2.101594 2.162030 0.386527
6 -3.220999 1.475281 0.925467
7 -2.581803 -0.796964 0.180331
6 -3.412540 0.082878 0.747753
6 -0.299269 -2.264241 -0.449077
1 5.304344 -0.163663 1.737743
1 5.382399 0.136858 -0.794636
1 3.185977 0.187888 -2.085134
1 0.932373 -0.311671 1.366224
1 3.017555 -0.393258 2.837678
1 -2.114644 3.263364 0.463786
1 -4.007715 2.050042 1.415626
1 -4.379471 -0.313239 1.099097
1 -0.572811 -2.828718 0.461055
1 0.789786 -2.379489 -0.603095
1 -0.795666 -2.747919 -1.311858
6 -3.146815 -2.155894 0.046938
1 -2.990568 -2.540510 -0.972499
1 -2.672661 -2.865421 0.746200
1 -4.233217 -2.149944 0.247135
6 -0.086130 2.536630 -0.792152
1 0.886270 2.480474 -0.265799
1 0.102603 2.306402 -1.853394
1 -0.445050 3.580750 -0.720938
Items in the first column are the only things that can be changed. Items in the other three columns should not ever change.
Thanks for your help!
the problem is the escaping of the "\"-character. Experiment with "\\s" or even "\\\\s". If you don't run the script directly, the "\"-character is evaluated two times: at first by the shell and then by awk. Anyway, you solution is way too complicated.
Try:
sed -i "s/^28 /METAL/" file
sed -i means substitute in place, so you don't have to copy the file "file" to "file.old" and then back again to "file".
Zsh has a built-in function to escape strings:
f="to be escaped"
print ${(q)f}
HTH Chris
If you can't win and quoting hell drives you mad (and you know there's a space and not a tab), just cheat:
awk '/^28 / { gsub(/^28 /, "METAL ") }; { print }' $file
... or else use [[:space:]] instead of \s, it appears GNU awk doesn't understand \s. For me, even plain
[0 1047 19:39:10] ~/temp/stack % gawk '/^28\s/ { gsub(/28\s/, "METAL") }; { print }' data
fails to replace. (Also, don't replace your space away if it's the only thing separating columns 1 and 2: replace with "METAL " or replace just /^28/.