I am new to bash /mysql however I have found tons of help reading threw examples and other people's problems ...but have ran into one of my own .
I am attempting to insert a row into an MySQL database table every time a file is added to a specific directory (for this i am using inotifywait ) anyways here is my bash script
#!/bin/bash
while true; do
filename= "false"
filename= inotifywait --format "%f" -e create /var/www/media2net/torrent
date_field= date +"%F":"%T"
mysql --host=localhost --user=root --password=admin Media2net << EOF
insert into video (title, description, url_video, upload_date)
values('testing','default_description','$filename', '$date_feild');
EOF
echo $filename
done
From this I have verified with echo the variable $filename is properly held at end of bash script however when i look at entry in the table the column url_video has it's default value and not the string represented by $filename
From what i can conclude the variable $filename does not get passed through EOF
i have tried as indicate here http://www.dba-oracle.com/t_access_mysql_with_bash_shell_script.htm
as well as this
Using shell script to insert data into remote MYSQL database
any help of where i can find how to pass variable into query would be greatly appreciated
In your example, filename is set to the empty string (mind the spaces after the = sign!). You need
filename=$(inotifywait --format "%f" -e create /var/www/media2net/torrent)
Similarly,
date_field=$(date +"%F:%T")
and be careful, you have a typo in your mysql command (date_field and note date_feild):
mysql --host=localhost --user=root --password=admin Media2net <<EOF
insert into video (title, description, url_video, upload_date)
values('testing','default_description','$filename', '$date_field');
EOF
Now I hope that you're controlling the filenames. Imagine a filename that contains a single quote e.g., hello'howdy. You'll have a problem in your query. Worse, an evil user who puts a file named whatever','something'); evil_mysql_command; whatever, you'll have the evil command performed! One possibility is to sanitize the filename using printf thus:
printf -v filename '%q' "$(inotifywait --format "%f" -e create /var/www/media2net/torrent)"
This will at least escape the single quotes that could appear in a filename. See Gordon Davisson's comment: the printf trick will not prevent from all the possible attacks, so I really hope you absolutely control the name of the files!
All these suggestions yield the following script:
#!/bin/bash
while true; do
printf -v filename '%q' "$(inotifywait --format "%f" -e create /var/www/media2net/torrent)"
date_field=$(date +"%F:%T")
mysql --host=localhost --user=root --password=admin Media2net <<EOF
insert into video (title, description, url_video, upload_date)
values('testing','default_description','$filename', '$date_field');
EOF
echo "$filename"
done
Edit.
To answer your question in the comment:
why did the script properly echo $filename to my terminal but not send it properly to MySQL, does that have to do with string starting with a space? or something else completely?
That's because when you do something like:
whatever= command
then the variable whatever is set to the empty string, and the command command is executed (with the whatever variable set as environment variable). E.g.,
$ IFS='c' read a b c <<< "AcBcC"
$ echo "$a $b $c"
A B C
$ echo $IFS
$
In your script, the variable filename was in fact never globally set. You can check it by doing this:
$ filename= "false"
$ echo "$filename"
$
What happens is that the environment variable filename is set to empty string then the command false (which happens to exist) is launched using that environment variable and we're done.
When you do this:
filename= inotifywait --format "%f" -e create /var/www/media2net/torrent
the variable filename is set to the empty string, and then the command inotifywait ... is executed with filename as an environment variable (but inotifywait doesn't really care about it). And that's what you saw on your terminal! it was the output of this command. Then you probably saw an empty line, that was the output of
echo $filename
which was equivalent to
echo
since the variable filename expanded to an empty string.
Hope this helps.
Related
I am trying to take the output from a MySQL query in bash and use it in a bash variable, but it keeps coming up blank when used in the script, but works perfectly from the terminal. What's wrong here?
I've tried changing the way the statement is written and changing the name of the variable just in case it was somehow reserved. I've also done a significant amount of searching but it turns out if you but 'bash', 'blank', and 'variable' in the search it usually comes up with some version of how to test for blank variables which I already know how to do.
tempo=$(mysql -u "$dbuser" -p"$dbpass" -D "$database" -t -s -r -N -B -e "select user from example where user='$temp' > 0;")
printf "the output should be: $tempo" # This is a test statement
The end result should be that the $tempo variable should either contain a user name from the database or be blank if there isn't one.
I think there is some error with your sql statement at user = '$temp' > 0.
But to get the result from MySql you have to redirect the standard error (stderr) to the standard output (stdout), you should use 2>&1.
Most probably you will run into MySql error but try running this on terminal.
tempo=$((mysql -u "$dbuser" -p"$dbpass" -D "$database" -t -s -r -N -B -e "select user from example where user='$temp' > 0;") 2>&1)
The solution was to echo the result of the sql query like this:
tempo=$(echo $(mysql -u "$dbuser" -p"$dbpass" -D "$database" -s -N -B -e "select user from example where user='$username' > 0;"))
Now I'm left with logic issues but I think I can handle that.
I maintain several servers having the same web app through bash scripts. Commands include basically ssh and scp, and they are the same for every server, just the IP, the user and the port are different between servers. So far I wrote as many commands as servers to maintain in same script. This works well but as I start to have many servers to maintain, I would prefer to list these servers in a MySQL table, and then use this list in my bash scripts, thus I I would not need to keep all of them updated when I have new servers to maintain.
Currently I have problems to extract data from MySQL in a proper way that could be then executed in bash script.
My current approach is to use CONCAT function in the query as follow:
outputofquery=$($MYSQL -u"$USER" --batch -p"$PASSWORD" --skip-column-names -h"$HOST" -e "SELECT CONCAT('scp -P ', server_port, ' $file ', server_user, '#', server_ip, ':/var/www/html/site/') FROM server_ssh;" $DBNAME)
echo ${outputofquery%$'\t;'*}
Giving the following result:
scp -P 22 text.php user1#1.1.1.1:/var/www/html/site/ scp -P 12345 text.php user2#2.2.2.2:/var/www/html/site/
Every command resulting from the MySQL query is put on the same line, meaning that this cannot be executed..
I though to add a semicolon just after site/ in the query, so that even if every command is on the same line, they could be executed independently but it happens that only the last scp command gets executed and those before are ignored.
Could you please let me know how I could execute ssh and scp commands in batch with data coming from a MySQL table?
Finally I succeed to execute batch commands with MySQL data. I build the commands to execute with concatenate function, depending on the line number extracted from MySQL. Below is my script
#!/bin/sh
HOST="localhost"
USER="root"
DBNAME="mydb"
PASSWORD="mypassord"
MYSQL="/Applications/XAMPP/bin/mysql"
file='text.php'
i=0;
command='';
ips_and_ports=$($MYSQL -u"$USER" -p"$PASSWORD" -h"$HOST" -BNe "SELECT server_port,server_user,server_ip FROM server_ssh;" $DBNAME)
for line in $ips_and_ports
do
(( i++ )); #increment the line number
if [ $(($i % 3)) -eq 1 ] #check if it is the first element of a line extracted from MySQL (server_port)
then
command="scp -P $line"; #initialize a new command to execute
elif [ $(($i % 3)) -eq 2 ] #check if it is the second element of a line extracted from MySQL (server_user)
then
command+=" $file $line#"; #concatenate
elif [ $(($i % 3)) -eq 0 ] #check if it is the third element of a line extracted from MySQL (server_ip)
then
command+="$line:/var/www/html/my_web_app/;"; #concatenate
eval $command; #execute the command
fi
done;
I can't put a comment because I'm new.
Have you tried to put a \n or && at the end of your mysql concatenation ?
You can also change the approach and assign the server and the port to variables and loop over these variables to execute the ssh and the scp command. For example
ips_and_ports=$($MYSQL -u"$USER" --batch -p"$PASSWORD" --skip-column-names -h"$HOST" -e "SELECT server_port, server_ip FROM extranet.server_ssh;" $DBNAME)
for $line in $ips_and_ports
do
OLDIFS=$IFS;
IFS=" "; # set the separator to the tab character (PS: this is 4 spaces, change accordingly )
for $el in $line
do
set -- $el; # separate each line by tab and assign the variables to $1, $2 ..
port=$1;
ip=$2;
scp -P 22 text.php user1#$ip:/var/www/html/site/ scp -P $port
done;
IFS=$OLDIFS; # put the separator back to it's original value
done;
I haven't tested the code but I hope you got my idea !
Good luck
more infos about the IFS and OLDIFS in link
I'm trying to import GZiped MySQL databases listed in a folder.
GZiped files are located at .mysqldumps/.
$NAME tries to extract database name (as files are always named database_name.sql.gz) and pass it to mysql command line.
Also, as username and database name are the same, the same argument is passed ($NAME).
As files are GZiped, we try to zcat them (so gunzip -c) before pipe them to mysql.
The full script is:
#!/bin/bash
FILES='.mysqldumps/*'
PASSWORD='MyPassword'
for f in $FILES
do
NAME=dbprefix_`basename $f .sql.gz`
echo "Processing $f"
set -x
zcat $f | mysql -u "$NAME" -p$PASSWORD "$NAME"
done
But, when i run the script it outputs:
./.mysqlimport
Processing .mysqldumps/first_database.sql.gz
+ mysql -u dbprefix_first_database -pMyPassword dbprefix_first_database
+ zcat .mysqldumps/first_database.sql.gz
ERROR 1044 (42000) at line 22: Access denied for user 'dbprefix_first_database'#'localhost' to database 'first_database'
As you can see, the selected database is 'first_database' instead of 'dbprefix_first_database' and this just trowns an error of corse, and i just can't understand why $NAME is not correctly parse as database name.
What i'm doing wrong?
After some investigation, the problem comes from the DUMP and not from the script.
While using mysqldump the option --databases was used which includes the USE 'dbname'; and when importing, that name was used instead of $NAME.
Problem solved!
I am trying to write a script that reads lines from a file containing:
a filename of an SQL script, and
some SQL queries on the same line
and launches a query.
There is a list.txt file in the following format:
query.sql; set #var:='prod';
where query.sql is a file with SQL queries, e.g.:
select *
from db
where system=#var
I am trying to write a Bash script that executes the queries like this:
mysql -uroot -proot -e
"set #var:='prod';
select *
from db
where system=#var;"
Here is what I have tried so far:
while read line;
do app=$(echo $line | awk '{for (i=2; i <= NF; i++) printf FS$i; print NL }';
cat `echo $line | awk -F\; '{print $1}'`; echo ";");
mysql -uroot -proot -e "`echo $app`"
done < list.txt
But I am having an SQL error, because the shell does not escape well the * character in the query.sql.
Debugging my code I obtain:
$echo $app
$set #var:='prod'; select a list.txt mysql query1.sql query.sql from db where system=#var ;
How can I adjust the code in order to have
$echo $app
$set #var:='prod'; select * from db where system=#var ;
?
Further details
There is a loop in the awk command, because the file list.txt may contain multiple queries, e.g.:
query.sql; set #var:='prod'; #a=asd; #b=zxc, #...=...;
I don't think you need to read the SQL files line by line. Assuming that there are no semicolons in the filenames, you can construct the query with the following AWK command:
awk -F\; '{
for (i = 2; i <= NF; i++) {
if ($i != "") printf("%s;", $i);
}
printf("source %s;\n", $1)
}' list.txt | mysql -uroot -proot
The command uses ; as a field separator. The for loop prints all fields starting from the second. The last printf function builds a query sourcing the SQL file.
The Cause of the Error
Your code fails to pass the correct SQL to the MySQL client because of this line:
mysql -uroot -proot -e "`echo $app`"
The shell interprets the special characters within the $app variable. In particular, the * character is expanded to "every file in the current directory". To prevent interpreting, use weak quoting, i.e. enclose the variable in double quotes: "$app":
When referencing a variable, it is generally advisable to enclose its name in double quotes. This prevents reinterpretation of all special characters within the quoted string -- except $, (backquote), and \ (escape)...
Keeping $ as a special character within double quotes permits referencing a quoted variable ("$variable"), that is, replacing the variable with its value
Also, there is no need for subshell expression (the backticks and the echo), if you want to get the value of the variable:
mysql -uroot -proot -e "$app"
So I need to read in a bunch of ID numbers from a file and do a mysql query on each of them. I started off with this:
#!/bin/bash
file="eidlist.txt"
while IFS= read -r line
do
mysql --host <redacted> --user <redacted> --password=<redacted> -N -e "use netops;select m_mailname from footprints where m_empno=$line;"
done <"$file"
This works and produces the expected output. Now I need to do the same thing but with a different list of IDs, querying a different field. Since the field I'm now querying on contains alphanumeric values (whereas the previous one was entirely numberic), I need to surround the value in quotes in the mysql query string, which I escape with \" like so:
#!/bin/bash
file="pidlist0.txt"
while IFS= read -r line
do
mysql --host <redacted> --user <redacted> --password=<redacted> -N -e "use netops;select m_mailname from footprints where m_stuid=\"$line\";"
done <"$file"
This doesn't work - the script produces no output. What am I doing wrong?
When I test the mysql command at the command line (with the $line variable pre-populated to one of the values from the file), it works, but when run from inside the script it produces no output. What's going on?
Just use m_stuid='$line';" ? Or change the double to single.