If in Loop in bash - mysql

This should be an incredibly easy question but I am not very familiar with bash and I am taking way longer than I should to figure it out.
declare -a ids=( 1 2 3 )
for i in "${ids[#]}";
do
re= $(mysql -h .... "SELECT col_A FROM DBA WHERE id=$i")
if [ $re -eq 0 ]; then
echo sucess
fi
done
This is an example of what I am trying to do, I have an id array and I want to send a query to my db so I can get a flag in the row with a certain id and then do something based on that. But I keep getting unexpected token errors and I am not entirely sure why
Edit: While copying the code and deleting some private information somehow I deleted the then, it was present in the code I was testing.

Based on what you described and the partial script, I am not certain I can completely create what you are trying to do but the token error messages you are experiencing usually have to do with the way bash handles whitespace as a delimiter. A few comments based on what you posted:
You need to remove the space around the equal sign in declaring an variable, so the space after the equal sign in re= needs to removed.
Because bash will is sensitive to whitespace, you need to quote variables declarations that might contain a space. To be safe, quotes need to be around the sub-shell $( )
You were missing the then in the if statement
It is important that variables in the test brackets, that is single [ ]s, must be quoted. Using an unquoted string with -eq, or even just the unquoted string alone within test brackets normally works, however, this is an unsafe practice and can give unpredictable results.
So, taking into account the items noted, the updated script would look something like:
declare -a ids=( 1 2 3 )
for i in "${ids[#]}";
do
re="$(mysql -h .... "SELECT col_A FROM DBA WHERE id=$i")"
if [ "$re" -eq "0" ]; then
echo "success"
fi
done
Can you try working the edits mentioned into your script and see if you are able to get it working? Remember, it will be helpful for you to use a site like ShellCheck to learn more about potential pitfalls or the uniquenesses of bash syntax. This will help to ensure you are working toward a solution to your specific need rather then getting trapped by some tricky syntax.
After you have worked through those edits, can you report back your experience?
EDIT
Based on your comments there is a good chance you are not running your script with bash despite the including #!/bin/bash at the top of your script. When you run the script as sh scriptname.sh you are forcing the script to be run by sh not bash. Try running your script like this /bin/bash scriptname.sh then report back on your experience.
For more information on the differences between various shells, see Unix/Linux : Difference between sh , csh , ksh and bash Shell

Your problem with your if statement is that you do not have the then keyword. A simple fix is:
declare -a ids=( 1 2 3 )
for i in "${ids[#]}";
do
re= $(mysql -h .... "SELECT col_A FROM DBA WHERE id=$i")
if [ $re -eq 0 ]; then
echo sucess
fi
done
Also here is a great reference on if statements in bash

Related

mysql query inside escaped bash -c string. How to put quotes?

Hello I am having trouble with a very specific line in a bash script.
Here is the code:
ssh $SOURCEIP "/usr/bin/time -f \"%e\" bash -c \"seq $ITER | parallel -n0 \"mysql --silent -h $TARGET -uroot -ppass -e 'SELECT * FROM dbname.tablename WHERE size = $SIZE;' >> out.txt\""
The problem is I ran out of quotes. The opening and escaped double quotes at the beginning of "mysql" are closing those from "bash -c". I have to put the mysql statement in double quotes and the query in single quotes, otherwise i get an error and I can't figure out how to proceed. I know that I should not pass the password like that and it will be changed later, I get this warning "$ITER"-times everytime i test this because --silent doesn't suppress this.
The problematic code is part of a small shell script that is supposed to just perform this data transfer.
I want to change to the other machine with ssh first and not via parallel because of consistency with other scripts.
So basically I need the double quotes around the bash -c command to get this whole parallel operation to work, which are already escaped because of the opening ssh doublequotes and also I need to put the mysql command inside quotes as well but they are closing each other somehow.
Any help will be greatly appreciated.
Thanks in advance.
Largio
Edit: (SOLUTION)
As suggested by #ole-tange the following command worked for me.
parallel --shellquote | parallel --shellquote
After invoking in a shell, i pasted my string in question into the prompt and got the masked string back. I still had troubles with finding out what exactly to paste but in the end it is just logical.
What exactly i pasted into the quoter was:
sql mysql://root:pass#$TARGET/ 'SELECT data FROM db_name.tablename WHERE size = ${SIZE};' >> out.txt
But still i had some problems with my variables inside my query. The problem here was that i had to de-mask the masking of the 2 variables $TARGET and $SIZE after everything got masked by the parallel quoter. Maybe my thinking has a too laborious manner but i could not get it to work in another way. Also note that i did not put quotes around the whole sql statement, as my plan was before, because now the quoter compensated for that. For consistency reasons i paste the final string that i got working in the end (with my changes afterwards):
ssh $SOURCEIP "/usr/bin/time -f \"%e\" bash -c \"seq $ITER | parallel -n0 sql\\\ mysql://root:pass#$TARGET/ \\\'SELECT\\\ data\\\ FROM\\\ db_name.tablename\\\ WHERE\\\ size\\\ =\\\ ${SIZE}\\\;\\\'\\\ \\\>\\\>\\\ out.txt\""
GNU Parallel has a quoter:
$ parallel --shellquote
"*\`$
[CTRL-D]
\"\*\\\`\$
And you can do it twice:
$ parallel --shellquote | parallel --shellquote
"*\`$
[CTRL-D]
\\\"\\\*\\\\\\\`\\\$
So just paste the string you want quoted.
But you might want to consider using functions and use env_parallel to copy the function:
myfunc() {
size=$1
target=$2
sql mysql://root:pass#$target/ "SELECT data FROM db_name.tablename WHERE size = $size;" >> out.txt
}
env_parallel --env myfunc -S $SOURCEIP --nonall myfunc $SIZE $TARGET
Also: Instead of mysql try sql mysql://root:pass#/ 'SELECT * FROM dbname.tablename WHERE size = $SIZE;'

Bash read line from variables

Hi & thanks in advance.
I'm trying to update a column(version) on an MySQL table from a Bash script.
I've populated a variable with the version numbers, but it fails after applying the first version in the list.
CODE:
UP_VER=`seq ${DB_VER} ${LT_VER} | sed '1d'`
UP_DB=`echo "UPDATE client SET current_db_vers='${UP_VER}' WHERE client_name='${CLIENT}'" | ${MYSQL_ID}`
while read -r line
do
${UP_DB}
if [[ "${OUT}" -eq "0" ]]; then
echo "Database upgraded.."
else
echo "Failed to upgrade.."
exit 1
fi
done < "${UP_VER}"
Thanks
Hopefully solved... My $UP_VER is in a a row not a column.
You're misunderstanding what several shell constructs do:
var=`command` # This executes the command immediately, and stores
# its result (NOT the command itself) in the variable
... < "${UP_VER}" # Treats the contents of $UP_VER as a filename, and tries
# to use that file as input
if [[ "${OUT}" -eq "0" ]]; then # $OUT is not defined anywhere
... current_db_vers='${UP_VER}' ... # this sets current_db_vers to the entire
# list of versions at once
Also, in the shell it's best to use lowercase (or mixed-case) variable names to avoid conflicts with the variables that have special meanings (which are all uppercase).
To fix the first problem, my recommendation is don't try to store shell commands in variables, it doesn't work right. (See BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail!.) Either use a function, or just write the command directly where it's going to be executed. In this case I'd vote for just putting it directly where it's going to be executed. BTW, you're making the same mistake with ${MYSQL_ID}, so I'd recommend fixing that as well.
For the second problem, you can use <<< "${UP_VER}" to feed a variable's contents as input (although this is a bashism, and not available in generic posix shells). But in this case I'd just use a for loop:
for ((ver=db_ver+1; ver<=lt_ver; ver++)); do
For the third problem, the simplest way to test the success of a command is to put it directly in the if:
if somecommand; then
echo "Database upgraded.."
else # ... etc
So, here's my take at a rewrite:
mysql_id() {
# appropriate function definition goes here...
}
for ((ver=db_ver+1; ver<=lt_ver; ver++)); do
if echo "UPDATE client SET current_db_vers='${ver}' WHERE client_name='${client}'" | mysql_id; then
echo "Database upgraded.."
else
echo "Failed to upgrade.."
exit 1
fi
done
... but I'm not sure I understand what it's supposed to do. It seems to be updating current_db_vers one number at a time until it reaches $ver_lt... but why not set it directly to $ver_lt in a single UPDATE?
try something like :
done <<< "${UP_VER}"

MySQL Procedure: substring_index throwing exception from special characters (executed in bash)

So the thing that makes this whole question hard is that I am working in a bash shell environment. I am parsing a large amount of data that is all located in text files in a set of directories. The environment I am working in does not have a gui, and is just the shell, and I am executing the commands from the shell through mysql, I am not logged into mysql.
I am the partner on a project, the main part is a bash script that searches for information and inserts it into text files in several directories. My operations parse out the needed data and inserts it into the database.
I run my main loop through a shell script. It loops through a set of directories and searches for the .txt files in each. I then pass the information to my procedure. In something like the below.
NOTE: I am not an expert in bash and have just started learning.
mysql - user -p'mypassword' --database=dbname <<EFO
call Procedure_Name("`cat ${textfile}`");
EOF
Since I am working in mysql and bash only I can not use another language to make my life easier so I use SUBSTRING_INDEX mostly. So an illustration of the procedure is shown below.
DELIMITER $$
CREATE PROCEDURE Procedure_name(textfile LONGTEXT)
BEGIN
DECLARE data LONGTEXT;
SET data = SUBSTRING_INDEX(SUBSTRING_INDEX(textfile,"(+++)",1),"(++)",-1));
INSERT INTO Table_Name (column) values (data);
END; $$
DELIMITER ;
The text file is a clean structure that allows for me to cut it up, but the problem I am having is that special characters inside of the textfile is causing my procedure to throw an error. I believe they are escape characters and I need a way around this. Just about any character could appear in the data I am parsing so I need a way to ignore these characters in the procedure or to cause them to not affect my process.
I tried looking into mysql_real_escape_string() however the parameters were hard to figure out and it looks like it only works in PHP but I am not sure. So I would like to do something at the beginning of my procedure to maybe insert "\"'s or something into the string to not cause my procedure to fail.
Also, these textfiles range from 16k to 11000k so I need something that can handle that. My process works sometimes but is getting caught up on a lot of stuff and my searching has not helped me at all. So any help would be greatly appreciated!!!
and thanks to all to reading this long description. normally I can find my answer or piece it together from questions but I had no luck this time so I figured it was about time to make an account and ask something.
Your question is really too board, but here is an example of what I mean
a script file:
#!/bin/bash
case $# in
1 ) inFile=$1 ;;
* ) echo "usage: myLoader infile"; exit 1 ;;
esac
awk 'BEGIN {
FS="\t"'; OFS="|"
}
{
sub(/badChars/, "", $0); sub(/otherBads/, "", $0) ; # .... as many as needed
# but be careful, easy to delete stuff that with too broad a brush.
print $1, $2, $5, $4, $9
}' $inFile > $inFile.psv
bcp -in -f ${formatFile:-formatFile} $inFile.psv
Note how awk makes it very easy, by repeating sub(...) commands to remove any "bad chars" you may have in your source data AND to reorganize the order of the columns in your data. Each $n is the value in numbered column on a line, so $1, $2, $5 skips fields $3 and $4, for example.
The OFS is set to the pipe char, making it easy to see in your output where exactly the field boundaries are AND if there are any leading or trailing whitespace characters that may be throwing off your load.
The > $inFile.psv keeps your original file, just in case you make a mistake in the awk script.
If you create really small test data files, you can eliminate saving to a file and just let the output go to the screen, editing until you get it right.
You'll have to find out exactly how mySQL's equivalent of bcp works. I'm pretty sure I've seen postings here. Either that, or post a separate question, "I have this pipe-delimited file with 8 columns, how do I load it to my table?".
The reference in my sample code to ${formatFile} is that hopefully the mySQL bcp command can take a format file that specifies the order and types of fields to be loaded into a file. Good bcp fmt files allow a fair amount of flexibility, but you'll have to read the man page for that utility AND do some research to understand the scope and restraints on that flexibility.
Going forward, you should post individual questions like, "I've tried x using lang Y to filter Z characters. Right now I'm getting output z, What am I doing wrong?"
Divide and conquer. There is no easy way. Reset those customer and boss expectations, you're learning something new, and it will take a little study to get it right. Good luck.
IHTH

Replacing output text of a command with a string in a Shell Script

Hello and thank you for any help you can provide
I have my Apache2 web server set up so that when I go to a specific link, it will run and display the output of a shell script stored on my server. I need to output the results of an SVN command (svn log). If I simply put the command 'svn log -q' (-q for quiet), I get the output of:
(of course not blurred), and with exactly 72 dashes in between each line. I need to be able to take these dashes, and turn them into an html line break, like so:
Basically I need the shell script to take the output of the 'svn log -q' command, search and replace every chunk of 72 dashes with an html line break, and then echo the output.
Is this at all possible?
I'm somewhat a noob at shell scripting, so please excuse any mess-ups.
Thank you so much for your help.
svn log -q | sed -e 's,-{72},<br/>,'
If you want to write it in the script this might help:
${string//substring/replacement}
Replace all matches of $substring with $replacement.
stringZ=abcABC123ABCabc
echo ${stringZ/abc/xyz} # xyzABC123ABCabc
# Replaces first match of 'abc' with 'xyz'.
echo ${stringZ//abc/xyz} # xyzABC123ABCxyz
# Replaces all matches of 'abc' with # 'xyz'.

Processing MySQL result in bash

I'm currently having a already a bash script with a few thousand lines which sends various queries MySQL to generate applicable output for munin.
Up until now the results were simply numbers which weren't a problem, but now I'm facing a challenge to work with a more complex query in the form of:
$ echo "SELECT id, name FROM type ORDER BY sort" | mysql test
id name
2 Name1
1 Name2
3 Name3
From this result I need to store the id and name (and their respective association) and based on the IDs need to perform further queries, e.g. SELECT COUNT(*) FROM somedata WHERE type = 2 and later output that result paired with the associated name column from the first result.
I'd know easily how to do it in PHP/Ruby , but I'd like to spare to fork another process especially since it's polled regularly, but I'm complete lost where to start with bash.
Maybe using bash is the wrong approach anyway and I should just fork out?
I'm using GNU bash, version 3.2.39(1)-release (i486-pc-linux-gnu).
My example is not Bash, but I'd like to point out my parameters at invoking the mysql command, they surpress the boxing and the headers.
#!/bin/sh
mysql dbname -B -N -s -e "SELECT * FROM tbl" | while read -r line
do
echo "$line" | cut -f1 # outputs col #1
echo "$line" | cut -f2 # outputs col #2
echo "$line" | cut -f3 # outputs col #3
done
You would use a while read loop to process the output of that command.
echo "SELECT id, name FROM type ORDER BY sort" | mysql test | while read -r line
do
# you could use an if statement to skip the header line
do_something "$line"
done
or store it in an array:
while read -r line
do
array+=("$line")
done < <(echo "SELECT id, name FROM type ORDER BY sort" | mysql test)
That's a general overview of the technique. If you have more specific questions post them separately or if they're very simple post them in a comment or as an edit to your original question.
You're going to "fork out," as you put it, to the mysql command line client program anyhow. So either way you're going to have process-creation overhead. With your approach of using a new invocation of mysql for each query you're also going to incur the cost of connecting to and authenticating to the mysqld server multiple times. That's expensive, but the expense may not matter if this app doesn't scale up.
Making it secure against sql injection is another matter. If you prompt a user for her name and she answers "sally;drop table type;" she's laughing and you're screwed.
You might be wise to use a language that's more expressive in the areas that are important for data-base access for some of your logic. Ruby, PHP, PERL are all good choices. PERL happens to be tuned and designed to run snappily under shell script control.