I'm trying to get values from a MySQL database, for a later substitution with perl, everything works fine but if a record contain space " " the variables will be set uncorrectly.
Example if in the database I have:
sede = "street"
fede = "calvin and hobbes"
lede = "12"
the result for the variables will be:
$TAGSEDE = "steet"
$TAGFEDE = "calvin"
$TAGLEDE = "and"
I understood there is something wrong in setting $DBDATAF but, I can't identify it (english isn't my mother language so some misunderstanding are more than a possibility).
DBDATAF=$(mysql -u$DBUSER -p$DBPASS -se "USE $DBNAME; SELECT sede, fede, lede FROM $DBTABL WHERE cf='$CFPI'")
read TAGSEDE TAGFEDE TAGLEDE <<< $DBDATAF
/usr/bin/perl -p -i -e "s/TAGDATAINSERT/$TAGDATAINSERT/g" $i
Your code breaks down for two reasons:
When $DBDATAF is passed to the <<< operator, the tabs are discarded. To keep them, double quote the variable as shown below.
read separates the input line into words using the IFS special variable. By default, it separates on tab, space or newline. So even though tabs are now preseved when passed to <<<, the read command splits on spaces also. Setting IFS to a tab makes read split as desired. Putting the IFS assignment and read on one line ensures that IFS returns to the default after read exits
IFS="$( echo -e '\t' )" read TAGSEDE TAGFEDE TAGLEDE <<< "$DBDATAF"
Related
General
I am trying to recursively search through hundreds of JSON files under a specific directory for lines that match a specific regular expression.
grep -rh works great for searching recursively for specific lines. I am having a problem applying a regular expression with the search because all the lines in the JSON files begin with a " and end in either ", or ".
Example: If I want to apply a regular expression to get all the lines that begin with zxc I will not be able to do it because the lines actually begin with "zxc
Code
The following command would work if the lines had no " at the beginning.
/bin/grep -rh -E "^(zxc)" "/etc/json_dir/"
The following command works, but I do not want grep to get hundreds of thousands of lines from all the JSON files and then apply a regular expression afterwards.
/bin/grep -rh -E ".*" "/etc/json_dir/" | /bin/sed -e 's/^"//g' -e 's/,$//g' -e 's/"$//g' | /bin/grep -E "^(zxc)"
Question
Is there a way for grep to ignore the " character at the beginning and " and ", characters at the end of the lines before it applies a regular expression ?
If there's no way, is there a way to do it with some other bash command, perl, python or some other language.
You can go with awk if I understand Your question properly:
awk '{gsub(/^"|"$/,"") } # this part removes all the "s from the start and end of line
/^WHAT/ { print } # or any other processing
' **/*.json
Note: the **/* requires the globestar recursive globbing option in (modern) bash.
See it in action at Ideone.
You can shorten it somewhat to:
awk '/^"?WHAT/' **/* # this executes the default printing action
But awk|sed|grep might not be the right tool to search JSON.
I am navigating a Java-based CLI menu on a remote machine with expect inside a bash script and I am trying to extract something from the output without leaving the expect session.
Expect command in my script is:
expect -c "
spawn ssh user#host
expect \"#\"
send \"java cli menu command here\r\"
expect \"java cli prompt\"
send \"java menu command\"
"
###I want to extract a specific string from the above output###
Expect output is:
Id Name
-------------------
abcd 12 John Smith
I want to extract abcd 12 from the above output into another expect variable for further use within the expect script. So that's the 3rd line, first field by using a double-space delimiter. The awk equivalent would be: awk -F ' ' 'NR==3 {$1}'
The big issue is that the environment through which I am navigating with Expect is, as I stated above, a Java CLI based menu so I can't just use awk or anything else that would be available from a bash shell.
Getting out from the Java menu, processing the output and then getting in again is not an option as the login process lasts for 15 seconds so I need to remain inside and extract what I need from the output using expect internal commands only.
You can use regexp in expect itself directly with the use of -re flag. Thanks to Donal on pointing out the single quote and double quote issues. I have given solution using both ways.
I have created a file with the content as follows,
Id Name
-------------------
abcd 12 John Smith
This is nothing but your java program's console output. I have tested this in my system with this. i.e. I just simulated your program's output with cat. You just replace the cat code with your program commands. Simple. :)
Double Quotes :
#!/bin/bash
expect -c "
spawn ssh user#domain
expect \"password\"
send \"mypassword\r\"
expect {\\\$} { puts matched_literal_dollar_sign}
send \"cat input_file\r\"; # Replace this code with your java program commands
expect -re {-\r\n(.*?)\s\s}
set output \$expect_out(1,string)
#puts \$expect_out(1,string)
puts \"Result : \$output\"
"
Single Quotes :
#!/bin/bash
expect -c '
spawn ssh user#domain
expect "password"
send "mypasswordhere\r"
expect "\\\$" { puts matched_literal_dollar_sign}
send "cat input_file\r"; # Replace this code with your java program commands
expect -re {-\r\n(.*?)\s\s}
set output $expect_out(1,string)
#puts $expect_out(1,string)
puts "Result : $output"
'
As you can see, I have used {-\r\n(.*?)\s\s}. Here the braces prevent any variable substitutions. In your output, we have a 2nd line with full of hyphens. Then a newline. Then your 3rd line content. Let's decode the regex used.
-\r\n is to match one literal hyphen and a new line together. This will match the last hyphen in the 2nd line and the newline which in turn make it to 3rd line now. So, .*? will match the required output (i.e. abcd 12) till it encounters double space which is matched by \s\s.
You might be wondering why I need parenthesis which is used to get the sub-match patterns.
In general, expect will save the expect's whole match string in expect_out(0,string) and buffer all the matched/unmatched input to expect_out(buffer). Each sub match will be saved in subsequent numbering of string such as expect_out(1,string), expect_out(2,string) and so on.
As Donal pointed out, it is better to use single quote's approach since it looks less messy. :)
It is not required to escape the \r with the backslash in case of double quotes.
Update :
I have changed the regexp from -\r\n(\w+\s+\w+)\s\s to -\r\n(.*?)\s\s.
With this way - your requirement - such as match any number of letters and single spaces until you encounter first occurrence of double spaces in the output
Now, let's come to your question. You have mentioned that you have tried -\r\n(\w+)\s\s. But, there is a problem here with \w+. Remember \w+ will not match space character. Your output has some spaces in it till double spaces.
The use of regexp will matter based on your requirements on the input string which is going to get matched. You can customize the regular expressions based on your needs.
Update version 2 :
What is the significance of .*?. If you ask separately, I am going to repeat what you commented. In regular expressions, * is a greedy operator and ? is our life saver. Let us consider the string as
Stackoverflow is already overflowing with number of users.
Now, see the effect of the regular expression .*flow as below.
* matches any number of characters. More precisely, it matches the longest string possible while still allowing the pattern itself to match. So, due to this, .* in the pattern matched the characters Stackoverflow is already over and flow in pattern matched the text flow in the string.
Now, in order to prevent the .* to match only up to the first occurrence of the string flow, we are adding the ? to it. It will help the pattern to behave as non-greedy manner.
Now, again coming back to your question. If we have used .*\s\s, then it will match the whole line since it is trying to match as much as possible. This is common behavior of regular expressions.
Update version 3:
Have your code in the following way.
x=$(expect -c "
spawn ssh user#host
expect \"password\"
send \"password\r\"
expect {\\\$} { puts matched_literal_dollar_sign}
send \"cat input\r\"
expect -re {-\r\n(.*?)\s\s}
if {![info exists expect_out(1,string)]} {
puts \"Match did not happen :(\"
exit 1
}
set output \$expect_out(1,string)
#puts \$expect_out(1,string)
puts \"Result : \$output\"
")
y=$?
# $x now contains the output from the 'expect' command, and $y contains the
# exit status
echo $x
echo $y;
If the flow happened properly, then exit code will have value as 0. Else, it will have 1. With this way, you can check the return value in bash script.
Have a look at here to know about the info exists command.
How to select mysql query with bash so each column will be in a separate array value?
I've tried the following command but it only works if the content is one word. for example:
id= 11, text=hello, important=1
if I've an article for instance in text. the code will not work properly. I guess I can use cut -f -d but if "text" contains special characters it wont work either.
while read -ra line; do
id=$(echo "${line[1]}")
text=$(echo "${line[2]}")
important=$(echo "${line[3]}")
echo "id: $id"
echo "text: $text"
echo "important: $important"
done < <(mysql -e "${selectQ}" -u${user} -p${password} ${database} -h ${host})
Bash by default splits strings at any whitespace character. First you need a unique column identifier for your output, you can use mysql --batch to get tab-separated csv output.
From the MySQL man page:
--batch, -B
Print results using tab as the column separator, with each row on a new line. With this option, mysql does not use the history file.
Batch mode results in nontabular output format and escaping of special characters. Escaping may be disabled by using raw mode; see the description for the --raw option
You want the result to be escaped, so don't use --raw, otherwise a tab character in your result data will break the loop again.
To skip the first row (column names) you can use the option --skip-column-names in addition
Now you can walk through each line and split it by tab character.
You can force bash to split by tab only by overriding the IFS variable (Internal Field Separator) temporarily.
Example
# myread prevents collapsing of empty fields
myread() {
local input
IFS= read -r input || return $?
while (( $# > 1 )); do
IFS= read -r "$1" <<< "${input%%[$IFS]*}"
input="${input#*[$IFS]}"
shift
done
IFS= read -r "$1" <<< "$input"
}
# loop though the result rows
while IFS=$'\t' myread id name surname url created; do
echo "id: ${id}";
echo "name: ${name}";
echo "surname: ${surname}";
echo "url: ${url}";
echo "created: ${created}";
done < <(mysql --batch --skip-column-headers -e "SELECT id, name, surname, url, created FROM users")
myread function all credits to this answer by Stefan Kriwanek
Attention:
You need to be very careful with quotes and variable delimiters.
If you just echo $row[0] without the curly brackets, you will get the wrong result
EDIT
You still have a problem , when a column returns empty string because the internal field separator matches any amount of the defined char:
row1\t\trow3 will create an array [row1,row3] instead of [row1,,row3]
I found a very nice approach to fix this, updated the example above.
Also read can directly seperate the input stream into variables.
My script need to store in a structure the result of a query:
#!/bin/bash
user="..."
psw="..."
database="..."
query="select name, mail from t"
customStructure=$(mysql -u$user -p$psw $database -e "$query";)
I've no idea how store the array of {name, mail} from query result..
I need structure like this:
array=[ [name1,mail1] , [name2,mail2], ....., [nameN, mailN] ]
Is there a way to do this in bash?
Bash arrays are initialized like so:
myarray=("hi" 1 "2");
To capture the individual portions of output of a command into an array, we must loop through the output, adding it's results to the array. That can be done like so:
for i in `echo "1 2 3 4"`
do
myarray+=($i)
done
In your example, it looks like you wish to get the output of a MySQL command and store the parts of it's output lines into subarrays. I will show you how to capture lines into arrays, and given that, you should be able to figure out how to put subarrays into that yourself.
while read line
do
myarray+=("$line")
done < <(mysql -u${user} -p${psw} ${database} -e "${query}")
It's also worth mentioning that for this kind of MySQL operation, where you don't need output metadata (such as pretty formatting and table names), you can use MySQL's -B option to do 'batch output'.
Field level record can be accessed via read -a command and IFS is set to the empty string to prevent read from stripping leading and trailing whitespace from the line.
#!/bin/bash
user="..."
psw="..."
database="..."
query="select name, mail from t"
OIFS="$IFS" ; IFS=$'\n' ; oset="$-" ; set -f
while IFS="$OIFS" read -a line
do
echo ${line[0]}
echo ${line[1]}
done < <(mysql -u${user} -p${psw} ${database} -e "${query}")
PHP has mysql_real_escape_string() to correctly escape any characters that might cause problems. What is the best way to mimic this functionality for BASH?
Is there anyway to do prepared mysql statements using bash? This seems to be the best way.
Most of my variables won't (shouldn't) have special characters, however I give the user complete freedom for their password. It may include characters like ' and ".
I may be doing multiple SQL statements so I'll want to make a script that takes in parameters and then runs the statement. This is what I have so far:
doSQL.sh:
#!/bin/sh
SQLUSER="root"
SQLPASS="passwor339c"
SQLHOST="localhost"
SQL="$1"
SQLDB="$2"
if [ -z "$SQL" ]; then echo "ERROR: SQL not defined"; exit 1; fi
if [ -z "$SQLDB" ]; then SQLDB="records"; fi
echo "$SQL" | mysql -u$SQLUSER -p$SQLPASS -h$SQLHOST $SQLDB
and an example using said command:
example.sh:
PASSWORD=$1
doSQL "INSERT INTO active_records (password) VALUES ('$PASSWORD')"
Obviously this would fail if the password password contained a single quote in it.
In Bash, printf can do the escaping for you:
$ a=''\''"\;:#[]{}()|&^$#!?, .<>abc123'
$ printf -v var "%q" "$a"
$ echo "$var"
\'\"\\\;:#\[\]\{\}\(\)\|\&\^\$#\!\?\,\ .\<\>abc123
I'll leave it to you to decide if that's aggressive enough.
This seems like a classic case of using the wrong tool for the job.
You've got a lot of work ahead of you to implement the escaping done by mysql_real_escape_string() in bash. Note that mysql_real_escape_string() actually delegates the escaping to the MySQL library which takes into account the connection and database character sets. It's called "real" because its predecessor mysql_escape_string() did not take the character set into consideration, and could be tricked into injecting SQL.
I'd suggest using a scripting language that has a MySQL library, such as Ruby, Python, or PHP.
If you insist on bash, then use the MySQL Prepared Statements syntax.
There is no escape from the following construct, no matter what quotes you use:
function quoteSQL() {
printf "FROM_BASE64('%s')" "$(echo -n "$1" | base64 -w0 )"
}
PASSWORD=$1
doSQL "INSERT INTO active_records (password) VALUES ($(quoteSQL "$PASSWORD"));"
# I would prefer piping
printf 'INSERT INTO active_records (password) VALUES (%s);\n' $(quoteSQL "$PASSWORD") | doSQL
mysql_real_escape_string() of course only escapes a single string literal to be quoted, not a whole statement. You need to be clear what purpose the string will be used for in the statement. According to the MySQL manual section on string literals, for inserting into a string field you only need to escape single and double quotation marks, backslashes and NULs. However, a bash string cannot contain a NUL, so the following should suffice:
#escape for MySQL single string
PASSWORD=${PASSWORD//\\/\\\\}
PASSWORD=${PASSWORD//\'/\\\'}
PASSWORD=${PASSWORD//\"/\\\"}
If you will be using the string after a LIKE, you will also probably want to escape % and _.
Prepared statements are another possibility. And make sure you don't use echo -e in your bash.
See also https://www.owasp.org/index.php/SQL_Injection_Prevention_Cheat_Sheet
This will escape apostrophes
a=$(echo "$1" | sed s/"'"/"\\\'"/g)
Please note though that mysql_real_escape_string also escapes \x00, \n, \r, \, " and \x1a. Be sure to escape these for full security.
To escape \x00 for example:
a=$(echo "$1" | sed s/"\x00"/"\\\'"/g)
With a bit of effort you can probably escape these using one sed command.
Sure, why not just use the real thing?
A script, anywhere, such as
~/scripts/mysqli_real_escape.php
#!/bin/php
<?php
$std_input_data = '';
$mysqli = new mysqli('localhost', 'username', 'pass', 'database_name');
if( ftell(STDIN) !== false ) $std_input_data = stream_get_contents(STDIN);
if( empty($std_input_data) ) exit('No input piped in');
if( mysqli_connect_errno( ) ) exit('Could not connect to database');
fwrite ( STDOUT,
$mysqli->real_escape_string($std_input_data)
);
exit(0);
?>
Next, run from bash terminal:
chmod +x ~/script/mysqli_real_escape.php`
ln -s ~/script/mysqli_real_escape.php /usr/bin/mysqli_real_escape
All set! Now you can use mysqli_real_escape in your bash scripts!
#!/bin/bash
MyString="stringW##)*special characters"
MyString="$(printf "$MyString" | mysqli_real_escape )"
Note: From what I understand, command substitution using "$(cmd ..."$var")" is preferred over using backticks. However, as no further nesting would be needed either should be fine.
Further Note: When inside command substitution, "$(...)", a new quote context is created. This is why the quotes around variables do not screw up the string.
This is how I did it, where my-file.txt contains spaces, new lines and quotes:
IFS='' content=$(cat my-file.txt)
mysql <flags> -e "update table set column = $(echo ${content#Q} | cut -c 2-) where something = 123"
Here are a couple Bash functions I wrote, grouped into a library.
It provides methods for proper quoting/escaping strings and identifiers:
##### db library functions #####
# Executes SQL Queries on localhost's MySQL server
#
# #Env
# $adminDBUser: The database user
# $adminDBPassword: The database user's password
#
# #Params
# $#: Optional MySQL arguments
#
# #Output
# >&1: The MySQL output stream
db::execute() {
# Uncomment below to debug
#tee --append debug.sql |
mysql \
--batch \
--silent \
--user="${adminDBUser:?}" \
--password="${adminDBPassword:?}" \
--host=localhost \
"$#"
}
# Produces a quoted string suitable for inclusion in SQL statements.
#
# #Params
# $1: The string to bo quoted
#
# #Output
# >&1: The quoted identifier suitable for inclusion in SQL statements
db::quoteString() {
local -- string="${1:?}"
local -- bas64String && bas64String=$(printf %s "${string}" | base64)
db::execute <<< "SELECT QUOTE(FROM_BASE64('${bas64String}'));"
}
# Produces a quoted identifier suitable for inclusion in SQL statements.
#
# #Params
# $1: The identifier to bo quoted
#
# #Output
# >&1: The quoted identifier suitable for inclusion in SQL statements
db::quoteIdentifier() {
local -- identifier="${1:?}"
local -- bas64Identifier && bas64Identifier=$(printf %s "${identifier}" | base64)
db::execute <<< "SELECT sys.quote_identifier(FROM_BASE64('${bas64Identifier}'))"
}
This will work:
echo "John O'hara" | php -R 'echo addslashes($argn);'
To pass it to a variable:
name=$(echo "John O'hara" | php -R 'echo addslashes($argn);')