I'm writing a sh file to run it via cron jobs, but I have no idea about shell script, I want to get the count of rows from mysql query then divide it by 200, ceil the result and start a loop from 0 to that amount.
After a long search, I wrote this line to get the count of rows from mysql query, and this works fine.
total=`mysql database -uuser -ppassword -s -N -e "SELECT count(id) as total FROM users"`
but all what I get from Google doesn't help me to complete my work, I tested something like "expr" and "let" methods for math operations, but I don't know why not working.
Even examples of loop I found on Google not working.
Can you help me with this script?
I guess you are using Bash. Here is how you divide:
#!/usr/bin/env bash
# ^
# |
# |
# --> This should be at the top of your script to make sure you run Bash
value=$((total / 200)) # total => value returned from mysql
for ((i = 0; i <= $value; i++)); do
# your code here
done
Stackoverflow has an astounding number of examples. I recommend you taking a look at Bash documentation here: https://stackoverflow.com/tags/bash/info.
Related
Hello I am having trouble with a very specific line in a bash script.
Here is the code:
ssh $SOURCEIP "/usr/bin/time -f \"%e\" bash -c \"seq $ITER | parallel -n0 \"mysql --silent -h $TARGET -uroot -ppass -e 'SELECT * FROM dbname.tablename WHERE size = $SIZE;' >> out.txt\""
The problem is I ran out of quotes. The opening and escaped double quotes at the beginning of "mysql" are closing those from "bash -c". I have to put the mysql statement in double quotes and the query in single quotes, otherwise i get an error and I can't figure out how to proceed. I know that I should not pass the password like that and it will be changed later, I get this warning "$ITER"-times everytime i test this because --silent doesn't suppress this.
The problematic code is part of a small shell script that is supposed to just perform this data transfer.
I want to change to the other machine with ssh first and not via parallel because of consistency with other scripts.
So basically I need the double quotes around the bash -c command to get this whole parallel operation to work, which are already escaped because of the opening ssh doublequotes and also I need to put the mysql command inside quotes as well but they are closing each other somehow.
Any help will be greatly appreciated.
Thanks in advance.
Largio
Edit: (SOLUTION)
As suggested by #ole-tange the following command worked for me.
parallel --shellquote | parallel --shellquote
After invoking in a shell, i pasted my string in question into the prompt and got the masked string back. I still had troubles with finding out what exactly to paste but in the end it is just logical.
What exactly i pasted into the quoter was:
sql mysql://root:pass#$TARGET/ 'SELECT data FROM db_name.tablename WHERE size = ${SIZE};' >> out.txt
But still i had some problems with my variables inside my query. The problem here was that i had to de-mask the masking of the 2 variables $TARGET and $SIZE after everything got masked by the parallel quoter. Maybe my thinking has a too laborious manner but i could not get it to work in another way. Also note that i did not put quotes around the whole sql statement, as my plan was before, because now the quoter compensated for that. For consistency reasons i paste the final string that i got working in the end (with my changes afterwards):
ssh $SOURCEIP "/usr/bin/time -f \"%e\" bash -c \"seq $ITER | parallel -n0 sql\\\ mysql://root:pass#$TARGET/ \\\'SELECT\\\ data\\\ FROM\\\ db_name.tablename\\\ WHERE\\\ size\\\ =\\\ ${SIZE}\\\;\\\'\\\ \\\>\\\>\\\ out.txt\""
GNU Parallel has a quoter:
$ parallel --shellquote
"*\`$
[CTRL-D]
\"\*\\\`\$
And you can do it twice:
$ parallel --shellquote | parallel --shellquote
"*\`$
[CTRL-D]
\\\"\\\*\\\\\\\`\\\$
So just paste the string you want quoted.
But you might want to consider using functions and use env_parallel to copy the function:
myfunc() {
size=$1
target=$2
sql mysql://root:pass#$target/ "SELECT data FROM db_name.tablename WHERE size = $size;" >> out.txt
}
env_parallel --env myfunc -S $SOURCEIP --nonall myfunc $SIZE $TARGET
Also: Instead of mysql try sql mysql://root:pass#/ 'SELECT * FROM dbname.tablename WHERE size = $SIZE;'
I have a query that sends the results to an email. I would like not to send an email if the query has NO results. How can i do that ?
heres the code
mysql -umy -hmysql1.com -P2 -pmysq <<<" Select * from Data.data "| mail -aFrom:test#test.com -s 'test' test#gmail.com
Not every task can be done easily in a single command pipeline. Trying to force it into a one-liner can make it hard to code and hard to maintain.
Feel free to write some statements in a script:
result=`mysql -umy -hmysql1.com -P2 -pmysq -e " Select * from Data.data "`
if [ -n "$result" ]
then
echo "$result" | mail -aFrom:test#test.com -s 'test' test#gmail.com
fi
The -n test is for strings being nonzero length. Read http://linuxcommand.org/lc3_man_pages/testh.html for more details on that.
Re your comment:
The statements I showed above are things you could type at the command-line in bash. Bash supports variables and "if/then/else" constructs and a lot more.
Writing a bash script is easy. Anything you can type at the command-line can be in a file. Open a text editor and write the lines I showed above. Save the file. For example it could be called "mailmyquery.sh" (the .sh extension is only customary, it's not required).
Exit the text editor. Then run:
bash mailmyquery.sh
And it runs the statements in the file as if you had written them yourself at the command-line.
Voilà! You are now a shell script programmer!
This should be an incredibly easy question but I am not very familiar with bash and I am taking way longer than I should to figure it out.
declare -a ids=( 1 2 3 )
for i in "${ids[#]}";
do
re= $(mysql -h .... "SELECT col_A FROM DBA WHERE id=$i")
if [ $re -eq 0 ]; then
echo sucess
fi
done
This is an example of what I am trying to do, I have an id array and I want to send a query to my db so I can get a flag in the row with a certain id and then do something based on that. But I keep getting unexpected token errors and I am not entirely sure why
Edit: While copying the code and deleting some private information somehow I deleted the then, it was present in the code I was testing.
Based on what you described and the partial script, I am not certain I can completely create what you are trying to do but the token error messages you are experiencing usually have to do with the way bash handles whitespace as a delimiter. A few comments based on what you posted:
You need to remove the space around the equal sign in declaring an variable, so the space after the equal sign in re= needs to removed.
Because bash will is sensitive to whitespace, you need to quote variables declarations that might contain a space. To be safe, quotes need to be around the sub-shell $( )
You were missing the then in the if statement
It is important that variables in the test brackets, that is single [ ]s, must be quoted. Using an unquoted string with -eq, or even just the unquoted string alone within test brackets normally works, however, this is an unsafe practice and can give unpredictable results.
So, taking into account the items noted, the updated script would look something like:
declare -a ids=( 1 2 3 )
for i in "${ids[#]}";
do
re="$(mysql -h .... "SELECT col_A FROM DBA WHERE id=$i")"
if [ "$re" -eq "0" ]; then
echo "success"
fi
done
Can you try working the edits mentioned into your script and see if you are able to get it working? Remember, it will be helpful for you to use a site like ShellCheck to learn more about potential pitfalls or the uniquenesses of bash syntax. This will help to ensure you are working toward a solution to your specific need rather then getting trapped by some tricky syntax.
After you have worked through those edits, can you report back your experience?
EDIT
Based on your comments there is a good chance you are not running your script with bash despite the including #!/bin/bash at the top of your script. When you run the script as sh scriptname.sh you are forcing the script to be run by sh not bash. Try running your script like this /bin/bash scriptname.sh then report back on your experience.
For more information on the differences between various shells, see Unix/Linux : Difference between sh , csh , ksh and bash Shell
Your problem with your if statement is that you do not have the then keyword. A simple fix is:
declare -a ids=( 1 2 3 )
for i in "${ids[#]}";
do
re= $(mysql -h .... "SELECT col_A FROM DBA WHERE id=$i")
if [ $re -eq 0 ]; then
echo sucess
fi
done
Also here is a great reference on if statements in bash
I have 2 Bash scripts that go through a directory, extract ID3 info from MP3's and them import the tag info into a MySQL DB. It takes quite a while to finish running so I was hoping somebody could help me make the scripts a bit more efficient.
Scripts are as follows:
makeid3dbentry.sh
TRACK=$(id3info "$1" | grep '^=== TIT2' | sed -e 's/.*: //g')
ARTIST=$(id3info "$1" | grep '^=== TPE1' | sed -e 's/.*: //g')
ALBUM=$(id3info "$1" | grep '^=== TALB' | sed -e 's/.*: //g')
ALBUMARTIST=$(id3info "$1" | grep '^=== TPE2' | sed -e 's/.*: //g')
COLS='`artist`,`name`,`album`,`albumartist`,`filename`'
# Replace all: ${string//substring/replacement} to escape "
VALS='"'${ARTIST//\"/\\\"}'","'${TRACK//\"/\\\"}'","'${ALBUM//\"/\\\"}'","'${ALBUMARTIST//\"/\\\"}'","'${1}'"'
SETLIST='`artist`="'${ARTIST//\"/\\\"}'",`name`="'${TRACK//\"/\\\"}'",`album`="'${ALBUM//\"/\\\"}'",`albumartist`="'${ALBUMARTIST//\"/\\\"}'",`filename`="'${1}'"'
echo 'INSERT INTO `music` ('${COLS}') VALUES ('${VALS}') ON DUPLICATE KEY UPDATE '${SETLIST}';'
exit
That produces an INSERT statement like
INSERT INTO `music` (`artist`,`name`,`album`,`albumartist`,`filename`) VALUES ("1200 Micrograms","Ayahuasca","1200 Micrograms","1200 Micrograms","/mnt/sharedmedia/music/Albums/1200 Micrograms/1200 Micrograms [2002]/1-01 - 1200 Micrograms - Ayahuasca.mp3") ON DUPLICATE KEY UPDATE `artist`="1200 Micrograms",`name`="Ayahuasca",`album`="1200 Micrograms",`albumartist`="1200 Micrograms",`filename`="/mnt/sharedmedia/music/Albums/1200 Micrograms/1200 Micrograms [2002]/1-01 - 1200 Micrograms - Ayahuasca.mp3";
That is then called from the main update script:
updatemusicdb.sh
DIRFULLPATH="${1}"
DIRECTORY=$(basename "${DIRFULLPATH}")
SQLFILE="/var/www/html/scripts/sql/rebuilddb_${DIRECTORY}.sql"
find "${DIRFULLPATH}" -type f -iname "*.mp3" -exec /var/www/html/scripts/bash/makeid3dbentry.sh {} > "${SQLFILE}" \;
mysql --defaults-extra-file=/var/www/html/config/website.cnf --default-character-set=utf8 "website" < "${SQLFILE}"
Unfortunately I don't know Bash & the Linux environment well enough to see where bottlenecks are and how to improve these scripts. I would appreciate any advice on improving the scripts or even a different approach if it's better / faster.
You can avoid running id3info multiple times as suggested in a comment already.
You can also make makeid3dbentry.sh take multiple arguments with a simple for file in "${#}" around your code. Then you can run find with -exec yourscript.sh {} + (works like xargs). This way you greatly reduce the number of invocations of your script. I'd probably recommend just doing this whole thing as one script, though. You can just wrap the insert statement generation in a for loop using your find command (without the -exec parameter) and pipe this output to file.
Assuming your MySQL database is using InnoDB you can speed up insertion by telling MySQL to skip committing the data until the job is done (instead of doing it on every insert).
Insert START TRANSACTION; in the top of your SQLFILE and COMMIT; at the bottom.
See http://dev.mysql.com/doc/refman/5.6/en/commit.html , http://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-bulk-data-loading.html
I'm currently having a already a bash script with a few thousand lines which sends various queries MySQL to generate applicable output for munin.
Up until now the results were simply numbers which weren't a problem, but now I'm facing a challenge to work with a more complex query in the form of:
$ echo "SELECT id, name FROM type ORDER BY sort" | mysql test
id name
2 Name1
1 Name2
3 Name3
From this result I need to store the id and name (and their respective association) and based on the IDs need to perform further queries, e.g. SELECT COUNT(*) FROM somedata WHERE type = 2 and later output that result paired with the associated name column from the first result.
I'd know easily how to do it in PHP/Ruby , but I'd like to spare to fork another process especially since it's polled regularly, but I'm complete lost where to start with bash.
Maybe using bash is the wrong approach anyway and I should just fork out?
I'm using GNU bash, version 3.2.39(1)-release (i486-pc-linux-gnu).
My example is not Bash, but I'd like to point out my parameters at invoking the mysql command, they surpress the boxing and the headers.
#!/bin/sh
mysql dbname -B -N -s -e "SELECT * FROM tbl" | while read -r line
do
echo "$line" | cut -f1 # outputs col #1
echo "$line" | cut -f2 # outputs col #2
echo "$line" | cut -f3 # outputs col #3
done
You would use a while read loop to process the output of that command.
echo "SELECT id, name FROM type ORDER BY sort" | mysql test | while read -r line
do
# you could use an if statement to skip the header line
do_something "$line"
done
or store it in an array:
while read -r line
do
array+=("$line")
done < <(echo "SELECT id, name FROM type ORDER BY sort" | mysql test)
That's a general overview of the technique. If you have more specific questions post them separately or if they're very simple post them in a comment or as an edit to your original question.
You're going to "fork out," as you put it, to the mysql command line client program anyhow. So either way you're going to have process-creation overhead. With your approach of using a new invocation of mysql for each query you're also going to incur the cost of connecting to and authenticating to the mysqld server multiple times. That's expensive, but the expense may not matter if this app doesn't scale up.
Making it secure against sql injection is another matter. If you prompt a user for her name and she answers "sally;drop table type;" she's laughing and you're screwed.
You might be wise to use a language that's more expressive in the areas that are important for data-base access for some of your logic. Ruby, PHP, PERL are all good choices. PERL happens to be tuned and designed to run snappily under shell script control.