Hello I am having trouble with a very specific line in a bash script.
Here is the code:
ssh $SOURCEIP "/usr/bin/time -f \"%e\" bash -c \"seq $ITER | parallel -n0 \"mysql --silent -h $TARGET -uroot -ppass -e 'SELECT * FROM dbname.tablename WHERE size = $SIZE;' >> out.txt\""
The problem is I ran out of quotes. The opening and escaped double quotes at the beginning of "mysql" are closing those from "bash -c". I have to put the mysql statement in double quotes and the query in single quotes, otherwise i get an error and I can't figure out how to proceed. I know that I should not pass the password like that and it will be changed later, I get this warning "$ITER"-times everytime i test this because --silent doesn't suppress this.
The problematic code is part of a small shell script that is supposed to just perform this data transfer.
I want to change to the other machine with ssh first and not via parallel because of consistency with other scripts.
So basically I need the double quotes around the bash -c command to get this whole parallel operation to work, which are already escaped because of the opening ssh doublequotes and also I need to put the mysql command inside quotes as well but they are closing each other somehow.
Any help will be greatly appreciated.
Thanks in advance.
Largio
Edit: (SOLUTION)
As suggested by #ole-tange the following command worked for me.
parallel --shellquote | parallel --shellquote
After invoking in a shell, i pasted my string in question into the prompt and got the masked string back. I still had troubles with finding out what exactly to paste but in the end it is just logical.
What exactly i pasted into the quoter was:
sql mysql://root:pass#$TARGET/ 'SELECT data FROM db_name.tablename WHERE size = ${SIZE};' >> out.txt
But still i had some problems with my variables inside my query. The problem here was that i had to de-mask the masking of the 2 variables $TARGET and $SIZE after everything got masked by the parallel quoter. Maybe my thinking has a too laborious manner but i could not get it to work in another way. Also note that i did not put quotes around the whole sql statement, as my plan was before, because now the quoter compensated for that. For consistency reasons i paste the final string that i got working in the end (with my changes afterwards):
ssh $SOURCEIP "/usr/bin/time -f \"%e\" bash -c \"seq $ITER | parallel -n0 sql\\\ mysql://root:pass#$TARGET/ \\\'SELECT\\\ data\\\ FROM\\\ db_name.tablename\\\ WHERE\\\ size\\\ =\\\ ${SIZE}\\\;\\\'\\\ \\\>\\\>\\\ out.txt\""
GNU Parallel has a quoter:
$ parallel --shellquote
"*\`$
[CTRL-D]
\"\*\\\`\$
And you can do it twice:
$ parallel --shellquote | parallel --shellquote
"*\`$
[CTRL-D]
\\\"\\\*\\\\\\\`\\\$
So just paste the string you want quoted.
But you might want to consider using functions and use env_parallel to copy the function:
myfunc() {
size=$1
target=$2
sql mysql://root:pass#$target/ "SELECT data FROM db_name.tablename WHERE size = $size;" >> out.txt
}
env_parallel --env myfunc -S $SOURCEIP --nonall myfunc $SIZE $TARGET
Also: Instead of mysql try sql mysql://root:pass#/ 'SELECT * FROM dbname.tablename WHERE size = $SIZE;'
Related
This question already has answers here:
How can I store a command in a variable in a shell script?
(12 answers)
Closed 4 years ago.
These work as advertised:
grep -ir 'hello world' .
grep -ir hello\ world .
These don't:
argumentString1="-ir 'hello world'"
argumentString2="-ir hello\\ world"
grep $argumentString1 .
grep $argumentString2 .
Despite 'hello world' being enclosed by quotes in the second example, grep interprets 'hello (and hello\) as one argument and world' (and world) as another, which means that, in this case, 'hello will be the search pattern and world' will be the search path.
Again, this only happens when the arguments are expanded from the argumentString variables. grep properly interprets 'hello world' (and hello\ world) as a single argument in the first example.
Can anyone explain why this is? Is there a proper way to expand a string variable that will preserve the syntax of each character such that it is correctly interpreted by shell commands?
Why
When the string is expanded, it is split into words, but it is not re-evaluated to find special characters such as quotes or dollar signs or ... This is the way the shell has 'always' behaved, since the Bourne shell back in 1978 or thereabouts.
Fix
In bash, use an array to hold the arguments:
argumentArray=(-ir 'hello world')
grep "${argumentArray[#]}" .
Or, if brave/foolhardy, use eval:
argumentString="-ir 'hello world'"
eval "grep $argumentString ."
On the other hand, discretion is often the better part of valour, and working with eval is a place where discretion is better than bravery. If you are not completely in control of the string that is eval'd (if there's any user input in the command string that has not been rigorously validated), then you are opening yourself to potentially serious problems.
Note that the sequence of expansions for Bash is described in Shell Expansions in the GNU Bash manual. Note in particular sections 3.5.3 Shell Parameter Expansion, 3.5.7 Word Splitting, and 3.5.9 Quote Removal.
When you put quote characters into variables, they just become plain literals (see http://mywiki.wooledge.org/BashFAQ/050; thanks #tripleee for pointing out this link)
Instead, try using an array to pass your arguments:
argumentString=(-ir 'hello world')
grep "${argumentString[#]}" .
In looking at this and related questions, I'm surprised that no one brought up using an explicit subshell. For bash, and other modern shells, you can execute a command line explicitly. In bash, it requires the -c option.
argumentString="-ir 'hello world'"
bash -c "grep $argumentString ."
Works exactly as original questioner desired. There are two restrictions to this technique:
You can only use single quotes within the command or argument strings.
Only exported environment variables will be available to the command
Also, this technique handles redirection and piping, and other shellisms work as well. You also can use bash internal commands as well as any other command that works at the command line, because you are essentially asking a subshell bash to interpret it directly as a command line. Here's a more complex example, a somewhat gratuitously complex ls -l variant.
cmd="prefix=`pwd` && ls | xargs -n 1 echo \'In $prefix:\'"
bash -c "$cmd"
I have built command processors both this way and with parameter arrays. Generally, this way is much easier to write and debug, and it's trivial to echo the command you are executing. OTOH, param arrays work nicely when you really do have abstract arrays of parameters, as opposed to just wanting a simple command variant.
I have a query that sends the results to an email. I would like not to send an email if the query has NO results. How can i do that ?
heres the code
mysql -umy -hmysql1.com -P2 -pmysq <<<" Select * from Data.data "| mail -aFrom:test#test.com -s 'test' test#gmail.com
Not every task can be done easily in a single command pipeline. Trying to force it into a one-liner can make it hard to code and hard to maintain.
Feel free to write some statements in a script:
result=`mysql -umy -hmysql1.com -P2 -pmysq -e " Select * from Data.data "`
if [ -n "$result" ]
then
echo "$result" | mail -aFrom:test#test.com -s 'test' test#gmail.com
fi
The -n test is for strings being nonzero length. Read http://linuxcommand.org/lc3_man_pages/testh.html for more details on that.
Re your comment:
The statements I showed above are things you could type at the command-line in bash. Bash supports variables and "if/then/else" constructs and a lot more.
Writing a bash script is easy. Anything you can type at the command-line can be in a file. Open a text editor and write the lines I showed above. Save the file. For example it could be called "mailmyquery.sh" (the .sh extension is only customary, it's not required).
Exit the text editor. Then run:
bash mailmyquery.sh
And it runs the statements in the file as if you had written them yourself at the command-line.
Voilà! You are now a shell script programmer!
This should be an incredibly easy question but I am not very familiar with bash and I am taking way longer than I should to figure it out.
declare -a ids=( 1 2 3 )
for i in "${ids[#]}";
do
re= $(mysql -h .... "SELECT col_A FROM DBA WHERE id=$i")
if [ $re -eq 0 ]; then
echo sucess
fi
done
This is an example of what I am trying to do, I have an id array and I want to send a query to my db so I can get a flag in the row with a certain id and then do something based on that. But I keep getting unexpected token errors and I am not entirely sure why
Edit: While copying the code and deleting some private information somehow I deleted the then, it was present in the code I was testing.
Based on what you described and the partial script, I am not certain I can completely create what you are trying to do but the token error messages you are experiencing usually have to do with the way bash handles whitespace as a delimiter. A few comments based on what you posted:
You need to remove the space around the equal sign in declaring an variable, so the space after the equal sign in re= needs to removed.
Because bash will is sensitive to whitespace, you need to quote variables declarations that might contain a space. To be safe, quotes need to be around the sub-shell $( )
You were missing the then in the if statement
It is important that variables in the test brackets, that is single [ ]s, must be quoted. Using an unquoted string with -eq, or even just the unquoted string alone within test brackets normally works, however, this is an unsafe practice and can give unpredictable results.
So, taking into account the items noted, the updated script would look something like:
declare -a ids=( 1 2 3 )
for i in "${ids[#]}";
do
re="$(mysql -h .... "SELECT col_A FROM DBA WHERE id=$i")"
if [ "$re" -eq "0" ]; then
echo "success"
fi
done
Can you try working the edits mentioned into your script and see if you are able to get it working? Remember, it will be helpful for you to use a site like ShellCheck to learn more about potential pitfalls or the uniquenesses of bash syntax. This will help to ensure you are working toward a solution to your specific need rather then getting trapped by some tricky syntax.
After you have worked through those edits, can you report back your experience?
EDIT
Based on your comments there is a good chance you are not running your script with bash despite the including #!/bin/bash at the top of your script. When you run the script as sh scriptname.sh you are forcing the script to be run by sh not bash. Try running your script like this /bin/bash scriptname.sh then report back on your experience.
For more information on the differences between various shells, see Unix/Linux : Difference between sh , csh , ksh and bash Shell
Your problem with your if statement is that you do not have the then keyword. A simple fix is:
declare -a ids=( 1 2 3 )
for i in "${ids[#]}";
do
re= $(mysql -h .... "SELECT col_A FROM DBA WHERE id=$i")
if [ $re -eq 0 ]; then
echo sucess
fi
done
Also here is a great reference on if statements in bash
I am trying to write a GNU make call function (example below) which has multiple shell commands to execute, such that it can be called with different arguments.
shell_commands = $(shell echo $(1); ls -ltr $(2))
try:
$(call shell_commands,$(FILE1),$(FILE2))
1) Is above the correct way to write a call function with multiple commands? By using a semi-colon to separate them? To make it readable, I write my targets as shown below. Is there a similar way to write a call function?
shell_commands:
echo $(1)
ls -ltr $(2)
2) I get this error from make when I execute make -B try. It looks like it is trying to execute /home/user/file1. But why?
make: execvp: /home/user/file1: Permission denied
make: *** [try] Error 127
3) Is it possible to pass variable number of parameters to a call function? Like pass in just the second parameter and not the first one.
$(call shell_commands,,$(FILE2))
I tried google-ing, searching on SO, and looking on gnu.org but I did not get any solutions. Will appreciate any answers or pointers to any resources which document call function with multiple optional arguments and commands.
Question 1: No, this is not right. The shell make function should NEVER be used inside a recipe: the recipe is already running in the shell, so why would you run another shell? It's just confusing. Second, it's generally recommended to use && between multiple commands in a recipe, so that if the first command fails the entire command will immediately fail, rather than continuing on and perhaps succeeding. Of course, that is not always correct either, it depends on what you're trying to do.
Question 2: This happens because the shell make function is like backticks in the shell: it expands to the output printed by the shell command it runs. Your shell command that make runs is:
echo $(1); ls -ltr $(2)
(where, one assumes, $1 expands to /home/user/file1) which prints the string /home/user/file1. After the expansion, that string is substituted into the recipe and make tries to run that recipe, giving the error you see above.
You want this, most likely:
shell_commands = echo $(1) && ls -ltr $(2)
try:
$(call shell_commands,$(FILE1),$(FILE2))
Now the call expands to the actual text, not an invocation of make's shell function, and then that text is run as the recipe.
Question 3: Sure, just using empty parameters means that the variable $1 (in this case) expands to the empty string.
I am helping to create a solution for a community project to monitor river levels for free, in the community spirit. The eventual product of this effort will be a system that takes data from a river level probe, and generates a graph for an online community.
I am early on with the project, and as I am familiar with Bash am using this to take a text file from the probe containing the data and push it into a MySQL database. All has been going really nicely until I hit a stumbling block. A number of my database table columns are numbers and so to interact with these using the MySQL command line tool backticks have to be used. Unfortunately I want to take the output from a MySQL statement and put it into a variable, but I think due to the two sets of back ticks MySQL is not getting the right command.
I have fudged it with a nasty work around like so which works all be it inefficiently:
mysql -N -D $targetDatabase -e "select \`"$timeSample"\` from RiverDataDays where date="$dateOfFile";" >tmpValue
dbEntry=`cat tmpValue`
echo $dbEntry
But actually, what I want to do is push it straight into an variable like this:
dbEntry=`mysql -N -D $targetDatabase -e "select \`"$timeSample"\` from RiverDataDays where date="$dateOfFile";"`
echo $dbEntry
Try doing this :
dbEntry="$(printf "SELECT \140%s\140 FROM 'RiverDataDays' WHERE date = '%s';\n" "$timeSample" "$(<tmpValue )" | mysql -N -D "$targetDatabase")"
echo "$dbEntry"
or
dbEntry="$(printf "SELECT \`%s\` FROM 'RiverDataDays' WHERE date = '%s';\n" "$timeSample" "$(<tmpValue )" | mysql -N -D "$targetDatabase")"
echo "$dbEntry"
The backquote (`) is used in the old-style command substitution, e.g.
foo=`command`
The
foo=$(command)
syntax is recommended instead. Backslash handling inside $() is less surprising, and $() is easier to nest. See http://mywiki.wooledge.org/BashFAQ/082
\140
is the octal representation of a backtick, see
man ascii
Followed Sputnick's link:-
http://mywiki.wooledge.org/BashFAQ/082
And then revised code accordingly
dbEntry=$(mysql -N -D $targetDatabase -e "select \`"$timeSample"\` from RiverDataDays where date="$dateOfFile";")
echo $dbEntry
Brilliant forum - thank you very much :)