I am currently trying to access variables outside of a loop which fetches 2 columns of a MySQL table in Bash:
mysql dbnameplaceholder -N -uusernameplaceholder -ppasswordplaceholder -
h127.0.0.1 -se "SELECT value1,value2 FROM table1 WHERE
value1='placeholdervalue'" | while read -r value1 value2;
do
var_value1="$value1"
var_value2="$value2"
done
echo "$var_value1 $var_value2" >> /tmp/script_debug
but the output is empty.
I did read up on the scope of variables here http://mywiki.wooledge.org/BashFAQ/024 and changed my original code to the following:
var_value1="NULL"
var_value1="NULL"
while read -r value1 value2;
do
var_value1="$value1"
var_value2="$value2"
done < <(mysql dbnameplaceholder -N -uusernameplaceholder -ppasswordplaceholder -
h127.0.0.1 -se "SELECT value1,value2 FROM table1 WHERE
value1='placeholdervalue'")
echo "$var_value1 $var_value2" >> /tmp/script_debug
but that did not work either.
By the way if i echo inside the loop i get the expected results so there is no MySQL query issue. Im no Bash expert so im kind of lost here.
Related
I want to store the result of my SQL query in a variable in order to use it in another query in the "in" clause.
When I do it with an INT list I have found the solution but in the context of a varach I have not yet found it.
Do you have an idea?
For information, for the int I use:
listID=$(/usr/mysql-5.5.40/bin/mysql -u$USER -p$PASSWORD -h$HOST -P$PORT $SCHEMA -e "SELECT ID FROM MYTABLE where NAME="toto";"|sed '1d;:a;N;$!ba;s/\n/,/g' )
In this case I can use listID for
NBMESSAGE=$(/usr/mysql-5.5.40/bin/mysql -u$USER -p$PASSWORD -h$HOST -P$PORT $SCHEMAMESSAGE -e "SELECT count(*) FROM MESSAGE where AUTOR_ID in ($listID)"|sed '1d')
then, I do
echo " Nombre de message $NBMESSAGE "
Thanks
Edit:
I had test my requete and result is good and produce this , in the Database ID are VARCHAR:
ID
1
12
13
14
15
16
17
When I pass |sed '1d;:a;N;$!ba;s/\n/,/g' I obtain
1,12,13,14,15,16,17
But I want have:
'1','12','13','14','15','16','17'
When I manage to get this result I will pass it in another query which does a count. I would post it as well.
EDIT 2:
I Solve my problem with this command, I share it with you :
listID=$(/usr/mysql-5.5.40/bin/mysql -u$USER -p$PASSWORD -h$HOST -P$PORT $SCHEMA -e "SELECT ID FROM MYTABLE where NAME="toto";"|se
d '1d;:a;N;$!ba;s/\n/'"'"','"'"'/g;s/^/'"'"'/g;s/$/'"'"'/g')
In this cas when I do a
echo " $listID "
I obtain '1','2','3','4'
thanks
Since I do not have a MySQL database with your dataset, I built this simple script that prints the output you are expecting:
printoutput.bash
#!/bin/bash
#
echo "ID"
echo "1"
echo "12"
echo "13"
echo "14"
echo "15"
echo "16"
echo "17"
This second script processes the output from the script above:
#!/bin/bash
# Read the result of the command into an array
# If the command fails, \0 is returned. The RESULTS array will have only 1 empty element
IFS=$'\n' read -r -d '' -a RESULTS < <(./printoutput.bash && printf '\0')
# Remove the first element
unset 'RESULTS[0]'
# Just to check that the unset worked
declare -p RESULTS
echo "-----------------------------------------------------------"
# This loops through the array, adding ' before and ', after each value
result=$(printf "'%s'," "${RESULTS[#]}")
# Remove the last ,
finalresult="${result%?}"
echo "$finalresult"
The method I use is to get rid of the \n all together by sending the output to an array. The array is then printed, adding single quotes and a comma. Then the last comma is removed.
Running the second script gives me the output you wanted:
./secondscript.bash
declare -a RESULTS=([1]="1" [2]="12" [3]="13" [4]="14" [5]="15" [6]="16" [7]="17")
-----------------------------------------------------------
'1','12','13','14','15','16','17'
#!/bin/bash
fut=$(date -d "+14 days" +'%Y-%m-%d')
echo $fut
mysql -u user -ppassword base1 << "EOF"
INSERT INTO tableB SELECT * FROM tableA WHERE startdate < $fut;
....many lines of MySQL....
EOF
Mysql engine doesn't understand $fut as a variable. What syntax should be applied to make this work?
I have no problem with using a here-document, but I would like to suggest another possibility which is slightly more verbose, but has advantages when you may have to do more than just substituting variables : piping the output of a function.
#!/bin/bash
sql_statement()
{
local fut=$(date -d "+14 days" +'%Y-%m-%d')
echo "INSERT INTO tableB SELECT * FROM tableA WHERE startdate < $fut;"
echo "....many lines of MySQL...."
}
mysql -u user -ppassword base1 < <(sql_statement)
This is not required in this specific case, but allows the function to have additional logic (e.g. if blocks, loops...) to generate a more complex or variable SQL statement. As a general solution, it has nice advantages.
You need to remove the " from the EOF:
#!/bin/bash
fut=$(date -d "+14 days" +'%Y-%m-%d')
echo $fut
mysql -u user -ppassword base1 << EOF
INSERT INTO tableB SELECT * FROM tableA WHERE startdate < $fut;
....many lines of MySQL....
EOF
From bash docs:
No parameter substitution when the "limit string" is quoted or escaped.
Newbie here, please pardon any confusing wording that I use.
A common task I have is to take a list of names and do a MySQL query to look the names up in a table and see if they are "live" on our site.
Doing this one at a time, my SQL query works fine. I then wanted to do the query using a loop from a file listing multiple names. This works fine, too.
I added this query loop to my bash profile so that I can quickly do the task by typing this:
$ ValidOnSite fileName
This works fine, and I even added an usage statement for my process to remind myself of the syntax. Below is what I have that works fine:
validOnSite() {
if [[ "$1" == "" ]] || [[ "$1" == "-h" ]] || [[ "$1" == "--help" ]]; then
echo "Usage:"
echo " $ validOnSite [filename]"
echo " Where validOnSite uses specified file as variables in sql query:"
echo " SELECT name, active FROM dbDb WHERE name=lines in file"
else
cat $1 | while read line ; do hgsql -h genome-centdb hgcentral -Ne "select name, active from dbDb where name='$line'" ; done
fi
Using a file "list.txt" which contains:
nameA
nameB
I would then type:
validOnSite list.txt
and both entries in list.txt meet my query criteria and are found in sql. My results will be:
nameA 1
nameB 1
Note the "1" after each result. I assume this is some sort of "yes" status.
Now, I add a third name to my list.txt, one that I know is not a match in sql. Now list.txt contains:
nameA
nameB
foo
When I again run this command for my list with 3 rows:
validOnSite list.txt
My results are the same as when I used the 1st version of file.txt, and I cannot see which lines failed, I still only see which lines were a success:
nameA 1
nameB 1
I have been trying all kinds of things to add a nested if statement, something that says, "If $line is a match, echo "pass", else echo "fail."
I do not want to see a "1" in my results. Using file.txt with 2 matches and 1 non-match, I would like my results to be:
nameA pass
nameB pass
foo fail
Or even better, color code a pass with green and a fail with red.
As I said, newbie here... :)
Any pointers in the right direction would help. Here is my latest sad attempt, but I realize I may be going in a wrong direction entirely:
validOnSite() {
if [[ "$1" == "" ]] || [[ "$1" == "-h" ]] || [[ "$1" == "--help" ]]; then
echo "Usage:"
echo " $ validOnSite [filename]"
echo " Where validOnSite uses specified file as variables in sql query:"
echo " SELECT name, active FROM dbDb WHERE name=lines in file"
else
cat $1 | while read line ; do hgsql -h genome-centdb hgcentral -Ne "select name, active from dbDb where name='$line'" > /dev/null ; done
if ( "status") then
echo $line "failed"
echo $line "failed" >> outfile
else
echo $line "ok"
echo $line "ok" >>outfile
clear
cat outfile
fi
fi
If something looks crazy in my last attempt, it's because it is - I am just googling around and trying as many things as I can while trying to learn. Any help appreciated, I feel stuck after working on this for a long time, but I am excited to move forward and find a solution! I think there is something I'm missing about understanding stdout, and also confusion about nested if's.
Note: I do not need an outfile, but it's ok if one is needed to accomplish the goal. stdout result alone would suffice, and is preferred.
Note: hgssql is just the name of our MySQL server. The MySQL part works fine, I am looking for a better way to deal with my bash output, and I think there is something about stderr that I'm missing. I'm looking for a fairly simple answer as I'm a newbie!
I guess, by hgsql you mean some Mercurial extension that allows to perform MySQL queries. I don't know how hgsql works, but I know that MySQL returns only the matching rows. But in terms of shell scripting, the result is a string that may contain extra information even if the number of matched rows is zero. For example, some MySQL client may return the header or a string like "No rows found", although it is unlikely.
I'll show how it is done with the official mysql client. I'm sure you will manage to adapt hgsql with the help of its documentation to the following example.
if [ -t 1 ]; then
red_color=$(tput setaf 1)
green_color=$(tput setaf 2)
reset_color=$(tput sgr0)
else
red_color=
green_color=
reset_color=
fi
colorize_flag() {
local color
if [ "$1" = 'fail' ]; then
color="$red_color"
else
color="$green_color"
fi
printf '%s' "${color}${1}${reset_color}"
}
sql_fmt='SELECT IF(active, "pass", "fail") AS flag FROM dbDb WHERE name = "%s"'
while IFS= read -r line; do
sql=$(printf "$sql_fmt" "$line")
flag=$(mysql --skip-column-names dbname -e "$sql")
[ -z "$flag" ] && flag='fail'
printf '%-20s%s\n' "$line" "$(colorize_flag "$flag")"
done < file
The first block detects if the script is running in interactive mode by checking if the file descriptor 1 (standard output) is opened on a terminal (see help test). If it is opened in a terminal, the script considers that the script is running interactively, i.e. the standard output is connected to the user's terminal directly, but not via pipe, for example. For interactive mode, it assigns variables to the terminal color codes with the help of tput command.
colorize_flag function accepts a string ($1) and outputs the string with the color codes applied according to its value.
The last block reads file line by line. For each line builds an SQL query string (sql) and invokes mysql command with the column names stripped off the output. The output of the mysql command is assigned to flag by means of command substitution. If "$flag" is empty, it is assigned to 'fail'. The $line and the colorized flag are printed to standard output.
You can test the non-interactive mode by chaining the output via pipe, e.g.:
./script | tee -a
I must warn you that it is generally bad idea to pass the shell variables into SQL queries unless the values are properly escaped. And the popular shells do not provide any tools to escape MySQL strings. So consider running the queries in Perl, PHP, or any programming language that is capable of building and running the queries safely.
Also note that in terms of performance it is better to run a single query and then parse the result set in a loop instead of running multiple queries in a loop, with the exception of prepared statements.
I found a way to get to my solution by piecing together the few basic things that I know. Not elegant, but it works well enough for now. I created a file "[filename]Results" with the output:
nameA 1
nameB 1
I then cut out the "1"s and made a new file. I then did a comparison with "[fileName]results" to list.txt in order to see what lines exist in file.txt but do not exist in results.
Note: I have the following in my .zshrc file.
validOnSite() {
if [[ "$1" == "" ]] || [[ "$1" == "-h" ]] || [[ "$1" == "--help" ]]; then
echo "Usage:"
echo " $ validOnSite [filename]"
echo " Where validOnSite uses specified file as variables in sql query:"
echo " SELECT name, active FROM dbDb WHERE name=lines in file"
else
cat $1 | while read line ; do hgsql -h genome-centdb hgcentral -Ne "select name from dbDb where name='$line' and active='1'" >> $1"Pass"; done
autoload -U colors
colors
echo $fg_bold[magenta]Assemblies active on site${reset_color}
echo
cat $1"Pass"
echo
echo $fg_bold[red]Not active or not found on site${reset_color}
comm -23 $1 $1"Pass" 2> /dev/null
echo
echo
mv $1"Pass" ~cath/myFiles/validOnSiteResults
echo "Results file containing only active assemblies resides in ~cath/myFiles/validOnSiteResults"
fi
}
list.txt:
nameA
nameB
foo
My input:
validOnSite list.txt
My output:
Assemblies active on site (<--this font is magenta)
nameA
nameB
Not active or not found on site (<--this font is red)
foo
Results file containing only active assemblies resides in ~me/myFiles/validOnRRresults
I am trying to store MySQL result into a global bash array variable but I don't know how to do it.
Should I save the MySQL command result in a file and read the file line by line in my for loop for my other treatment?
Example:
user password
Pierre aaa
Paul bbb
Command:
$results = $( mysql –uroot –ppwd –se « SELECT * from users );
I want that results contains the two rows.
Mapfile for containing whole table into one bash variable
You could try this:
mapfile result < <(mysql –uroot –ppwd –se "SELECT * from users;")
Than
echo ${result[0]%$'\t'*}
echo ${result[0]#*$'\t'}
or
for row in "${result[#]}";do
echo Name: ${row%$'\t'*} pass: ${row#*$'\t'}
done
Nota This will work fine while there is only 2 fields by row. More is possible but become tricky
Read for reading table row by row
while IFS=$'\t' read name pass ;do
echo name:$name pass:$pass
done < <(mysql -uroot –ppwd –se "SELECT * from users;")
Read and loop to hold whole table into many variables:
i=0
while IFS=$'\t' read name[i] pass[i++];do
:;done < <(mysql -uroot –ppwd –se "SELECT * from users;")
echo ${name[0]} ${pass[0]}
echo ${name[1]} ${pass[1]}
New (feb 2018) shell connector
There is a little tool (on github) or on my own site: (shellConnector.sh you could use:
Some preparation:
cd /tmp/
wget -q http://f-hauri.ch/vrac/shell_connector.sh
. shell_connector.sh
newSqlConnector /usr/bin/mysql '–uroot –ppwd'
Following is just for demo, skip until test for quick run
Thats all. Now, creating temporary table for demo:
echo $SQLIN
3
cat >&3 <<eof
CREATE TEMPORARY TABLE users (
id bigint(20) unsigned NOT NULL PRIMARY KEY AUTO_INCREMENT,
name VARCHAR(30), date DATE)
eof
myMysql myarray ';'
declare -p myarray
bash: declare: myarray: not found
The command myMysql myarray ';' will send ; then execute inline command,
but as mysql wont anwer anything, variable $myarray wont exist.
cat >&3 <<eof
INSERT INTO users VALUES (1,'alice','2015-06-09 22:15:01'),
(2,'bob','2016-08-10 04:13:21'),(3,'charlie','2017-10-21 16:12:11')
eof
myMysql myarray ';'
declare -p myarray
bash: declare: myarray: not found
Operational Test:
Ok, then now:
myMysql myarray "SELECT * from users;"
printf "%s\n" "${myarray[#]}"
1 alice 2015-06-09
2 bob 2016-08-10
3 charlie 2017-10-21
declare -p myarray
declare -a myarray=([0]=$'1\talice\t2015-06-09' [1]=$'2\tbob\t2016-08-10' [2]=$'3\tcharlie\t2017-10-21')
This tool are in early step of built... You have to manually clear your variable before re-using them:
unset myarray
myMysql myarray "SELECT name from users where id=2;"
echo $myarray
bob
declare -p myarray
declare -a myarray=([0]="bob")
If you're looking to get a global variable inside your script you can simply assign a value to a varname:
VARNAME=('var' 'name') # no space between the variable name and value
Doing this you'll be able to access VARNAME's value anywhere in your script after you initialize it.
If you want your variable to be shared between multiple scripts you have to use export:
script1.sh:
export VARNAME=('var' 'name')
echo ${VARNAME[0]} # will echo 'var'
script2.sh
echo ${VARNAME[1]} # will echo 'name', provided that
# script1.sh was executed prior to this one
NOTE that export will work only when running scripts in the same shell instance. If you want it to work cross-instance you would have to put the export variable code somewhere in .bashrc or .bash_profile
The answer from #F. Hauri seems really complicated.
https://stackoverflow.com/a/38052768/470749 helped me realize that I needed to use parentheses () wrapped around the query result to treat is as an array.
#You can ignore this function since you'll do something different.
function showTbl {
echo $1;
}
MOST_TABLES=$(ssh -vvv -t -i ~/.ssh/myKey ${SERVER_USER_AND_IP} "cd /app/ && docker exec laradock_mysql_1 mysql -u ${DB} -p${REMOTE_PW} -e 'SELECT table_name FROM information_schema.tables WHERE table_schema = \"${DB}\" AND table_name NOT LIKE \"pma_%\" AND table_name NOT IN (\"mail_webhooks\");'")
#Do some string replacement to get rid of the query result header and warning. https://stackoverflow.com/questions/13210880/replace-one-substring-for-another-string-in-shell-script
warningToIgnore="mysql\: \[Warning\] Using a password on the command line interface can be insecure\."
MOST_TABLES=${MOST_TABLES/$warningToIgnore/""}
headerToIgnore="table_name"
MOST_TABLES=${MOST_TABLES/$headerToIgnore/""}
#HERE WAS THE LINE THAT I NEEDED TO ADD! Convert the string to array:
MOST_TABLES=($MOST_TABLES)
for i in ${MOST_TABLES[#]}; do
if [[ $i = *[![:space:]]* ]]
then
#Remove whitespace from value https://stackoverflow.com/a/3232433/470749
i="$(echo -e "${i}" | tr -d '[:space:]')"
TBL_ARR+=("$i")
fi
done
for t in ${TBL_ARR[#]}; do
showTbl $t
done
This successfully shows me that ${TBL_ARR[#]} has all the values from the query result.
results=($( mysql –uroot –ppwd –se "SELECT * from users" ))
if [ "$?" -ne 0 ]
then
echo fail
exit
fi
I'm trying to obtain a recordset and read a few fields from it. I'm not able to figure it out how put the fields into variables. The script is:
#!/bin/bash
sqlQuery="$(mysql -h host -u user -ppass -D oberonsaas_v2 -s -N -e
'select ventas.id_venta,
ventas_entradas.id_ventas_entradas,
ventas.id_evento,
id_tarifa,
DATE_FORMAT(fecha_evento,"%Y%m%d") as fecha,
TIME_FORMAT(pase,"%H%i") as pase
from pases,ventas,ventas_entradas,recintos
where ventas.id_recinto = recintos.id_recinto
and ventas.id_pase = pases.id_pase
and ventas.id_venta = ventas_entradas.id_venta
and recintos.id_cliente = 32
and ventas.estado="Pagada"
and date(fecha_venta) = date_add(date(CURRENT_TIMESTAMP),INTERVAL -1 day)')"
echo $sqlQuery
I get all the recordset in $sqlQuery, but i want to do a loop and concat the fields.
Well I have invented some output
#!/bin/bash
Result="21,336,purchase,tarif_exspensive,Jose"
(IFS=","
for i in $Result
do
echo "I Have: " $i
done
)
Ouput is
$ bash t5.sh
I Have: 21
I Have: 336
I Have: purchase
I Have: tarif_exspensive
I Have: Jose
Collapse this into the simplest possible construct.
Put returned recordset into a bash array (for example):
sqlQuery=( pases_value ventas_value ventas_entradas_value recintos_value )
note: yours could look like:
sqlQuery=( $(mysql -h host ... )
new lines between the array operators "(" and ")" are ok
each will be in a distinct sqlQuery array location.
for example return: recintos_value
echo ${sqlQuery[3]}
concat all:
echo ${sqlQuery[*]}
if delimiters are of concern, use the mysql --delimiter switch to define a desired character, then
set concatValue=${sqlQuery[*]}
then to remove the delimiter character ("," for example):
set finalValue=${concatValue//,}