I've the following bash script to upgrade my database schema. Script reads hostname and database password from command line.
The problem is here that if the password is alphanumeric e.g r00t then script works. But if password contains special characters e.g pa**w0rd, then script does not work and directly exits. Please help me with this. Thanks.
#!/bin/bash
echo "Enter hostname."
read -p "Hostname [localhost]: " DB_HOST
DB_HOST=${DB_HOST:-localhost}
echo "Enter MySQL root password"
DB_PASS=
while [[ $DB_PASS = "" ]]; do
read -sp "Password: " DB_PASS
done
MYSQL="mysql --force --connect-timeout=90 --host=$DB_HOST -u root --password=${DB_PASS}"
# Apply schema updates. My DBName is "mydb"
# Upgrade schema file is stored in "mysql" folder
$MYSQL mydb -e exit > /dev/null 2>&1 && $MYSQL mydb < "../mysql/upgrade_schema_v.2.1.sql"
Logging into mysql using bash
For ubuntu or linux shell try to use command
mysql -u username -p'p#ssw()rD'
for remote host login use
mysql -h hostname -u user -p'password'
This is occurring because you are using shell GLOB (wildcard) characters in the password, and in Bash (or on Linux generally) wildcards are expanded by the shell.
The safest and most reliable solution is to not use shell wildcard characters or other characters interpreted by the shell in the password. You should also avoid spaces. There are plenty of other characters.
Here are the ones you should avoid:
" ' $ , [ ] * ? { } ~ # % \ < > | ^ ;
Here are the ones it is usually safe to use:
: # . , / + - ! =
To ensure the password is still secure, make it longer. As an example:
K#3amvv7l1wz1192sjqhym
This meets old-fashioned password complexity rules, because upper, lower, numbers and special characters are in the first four, the remainder is randomly generated but avoids any problematic characters.
However if you must use them, you can quote the password parameter with single quotes - though you will still run in to trouble if the password contains single quotes!
Try enclosing your password in single quotes.
If it's pa**w0rd, use 'pa**w0rd'
Variables are best used for data, not code. The layers of variables make it hard to protect the expansion when you want some parts of the expansion (i.e., you want your command line to be word split on the tokens you want), but don't want all the other effects. The solution is to not store the code in a string. Instead, use a function like:
do_mysql() {
host="$1"
pass="$2"
mysql --force --connect-timeout=90 --host="$host" -u root --password="$pass" "$#"
}
then you can run it with extra arguments like
do_mysql "$DB_HOST" "$DB_PASS" -e exit > /dev/null && do_mysql "$DB_HOST" "$DB_PASS" < "../mysql/upgrade_schema_v.2.1.sql"
Though it would also be better not to use upper case for your variables. Doing so makes it so you could collide with environment variables and accidentally change things you don't intend to change (as the number of people who accidentally reset PATH can attest).
I have implemented a api from which I wanted to run docker commands. Most commands worked fine, but when I tried to run the following command
{
"command":"docker run -d -p 1506:1506 –p 2003:2003 --name=dradis dradis-licensed:v1"
}
on the server side the 2nd '-p' parameter was actually being converted to '?p'
Any clues why this would happen and any escape charaters I should be using?
Cheers
Kris
MySQL is awesome! I am currently involved in a major server migration and previously, our small database used to be hosted on the same server as the client. So we used to do this : SELECT * INTO OUTFILE .... LOAD DATA INFILE ....
Now, we moved the database to a different server and SELECT * INTO OUTFILE .... no longer works, understandable - security reasons I believe.
But, interestingly LOAD DATA INFILE .... can be changed to LOAD DATA LOCAL INFILE .... and bam, it works.
I am not complaining nor am I expressing disgust towards MySQL. The alternative to that added 2 lines of extra code and a system call form a .sql script. All I wanted to know is why LOAD DATA LOCAL INFILE works and why is there no such thing as SELECT INTO OUTFILE LOCAL?
I did my homework, couldn't find a direct answer to my questions above. I couldn't find a feature request # MySQL either. If someone can clear that up, that had be awesome!
Is MariaDB capable of handling this problem?
From the manual: The SELECT ... INTO OUTFILE statement is intended primarily to let you very quickly dump a table to a text file on the server machine. If you want to create the resulting file on some client host other than the server host, you cannot use SELECT ... INTO OUTFILE. In that case, you should instead use a command such as mysql -e "SELECT ..." > file_name to generate the file on the client host."
http://dev.mysql.com/doc/refman/5.0/en/select.html
An example:
mysql -h my.db.com -u usrname--password=pass db_name -e 'SELECT foo FROM bar' > /tmp/myfile.txt
You can achieve what you want with the mysql console with the -s (--silent) option passed in.
It's probably a good idea to also pass in the -r (--raw) option so that special characters don't get escaped. You can use this to pipe queries like you're wanting.
mysql -u username -h hostname -p -s -r -e "select concat('this',' ','works')"
EDIT: Also, if you want to remove the column name from your output, just add another -s (mysql -ss -r etc.)
The path you give to LOAD DATA INFILE is for the filesystem on the machine where the server is running, not the machine you connect from. LOAD DATA LOCAL INFILE is for the client's machine, but it requires that the server was started with the right settings, otherwise it's not allowed. You can read all about it here: http://dev.mysql.com/doc/refman/5.0/en/load-data-local.html
As for SELECT INTO OUTFILE I'm not sure why there is not a local version, besides it probably being tricky to do over the connection. You can get the same functionality through the mysqldump tool, but not through sending SQL to the server.
Since I find myself rather regularly looking for this exact problem (in the hopes I missed something before...), I finally decided to take the time and write up a small gist to export MySQL queries as CSV files, kinda like https://stackoverflow.com/a/28168869 but based on PHP and with a couple of more options. This was important for my use case, because I need to be able to fine-tune the CSV parameters (delimiter, NULL value handling) AND the files need to be actually valid CSV, so that a simple CONCAT is not sufficient since it doesn't generate valid CSV files if the values contain line breaks or the CSV delimiter.
Caution: Requires PHP to be installed on the server!
(Can be checked via php -v)
"Install" mysql2csv via
wget https://gist.githubusercontent.com/paslandau/37bf787eab1b84fc7ae679d1823cf401/raw/29a48bb0a43f6750858e1ddec054d3552f3cbc45/mysql2csv -O mysql2csv -q && (sha256sum mysql2csv | cmp <(echo "b109535b29733bd596ecc8608e008732e617e97906f119c66dd7cf6ab2865a65 mysql2csv") || (echo "ERROR comparing hash, Found:" ;sha256sum mysql2csv) ) && chmod +x mysql2csv
(download content of the gist, check checksum and make it executable)
Usage example
./mysql2csv --file="/tmp/result.csv" --query='SELECT 1 as foo, 2 as bar;' --user="username" --password="password"
generates file /tmp/result.csv with content
foo,bar
1,2
help for reference
./mysql2csv --help
Helper command to export data for an arbitrary mysql query into a CSV file.
Especially helpful if the use of "SELECT ... INTO OUTFILE" is not an option, e.g.
because the mysql server is running on a remote host.
Usage example:
./mysql2csv --file="/tmp/result.csv" --query='SELECT 1 as foo, 2 as bar;' --user="username" --password="password"
cat /tmp/result.csv
Options:
-q,--query=name [required]
The query string to extract data from mysql.
-h,--host=name
(Default: 127.0.0.1) The hostname of the mysql server.
-D,--database=name
The default database.
-P,--port=name
(Default: 3306) The port of the mysql server.
-u,--user=name
The username to connect to the mysql server.
-p,--password=name
The password to connect to the mysql server.
-F,--file=name
(Default: php://stdout) The filename to export the query result to ('php://stdout' prints to console).
-L,--delimiter=name
(Default: ,) The CSV delimiter.
-C,--enclosure=name
(Default: ") The CSV enclosure (that is used to enclose values that contain special characters).
-E,--escape=name
(Default: \) The CSV escape character.
-N,--null=name
(Default: \N) The value that is used to replace NULL values in the CSV file.
-H,--header=name
(Default: 1) If '0', the resulting CSV file does not contain headers.
--help
Prints the help for this command.
Using mysql CLI with -e option as Waverly360 suggests is a good one, but that might go out of memory and get killed on large results. (Havent find the reason behind it).
If that is the case, and you need all records, my solution is: mysqldump + mysqldump2csv:
wget https://raw.githubusercontent.com/jamesmishra/mysqldump-to-csv/master/mysqldump_to_csv.py
mysqldump -u username -p --host=hostname database table | python mysqldump_to_csv.py > table.csv
Re: SELECT * INTO OUTFILE
Check if MySQL has permissions to write a file to the OUTFILE directory on the server.
Try setting path to /var/lib/mysql-files/filename.csv (MySQL 8). Determine what files directory is yours by typping SHOW VARIABLES LIKE "secure_file_priv"; in mysql client command line.
See answer about here: (...) --secure-file-priv in MySQL answered in 2015 by vhu user
I'm trying to execute this:
$ mysql --user=XXX --password=XXX --batch --skip-column-names \
-e "SELECT userid, displayname FROM Users" stackoverflowdb | \
split -l 50 -a 5 - "result."
but bash complains about ) 'unexpected' character (i have this character and few other 'weird ones' in my mysql password). I tried to take my password in quotes --password="myweirdpasshere" but then mysql won't login me at all (probably password is incorrect?)
Some characters such as $, `, ", and sometimes \ retain their special meaning inside double quotes. If your password contains any of those characters, use single quotes instead. (Unless your password also contains single quotes, in which case you might just drop the quotes altogether and put a \ before each special character.)
You should be able to put the password in the $HOME/.my.cnf file, avoiding this issue as well as increasing the security.
Another option to using $HOME/.my.cnf is embedding the configuration file into the script. A comment on the documentation gives examples on this. For your case it should be:
$ mysql --user=XXX --defaults-file <(printf '[client]\npassword=XXX\n') \
--batch --skip-column-names \
-e "SELECT userid, displayname FROM Users" stackoverflowdb | \
split -l 50 -a 5 - "result."
If you don't give the password on the command line, it will bring up an interactive prompt. That might solve your error. Just use -p instead of --password=XXX.
Of course, if you require unattended access to the script, that's not a very useful answer. It would be more helpful if you could create a minimal example and tell us the exact error that Bash gives you.
I'm trying to execute this mysql command from a batch file:
mysql -f -utest -ppass db < alter1.sql
However, < is an escape character. I tried nesting it in double-quotes, but the double quotes end up appearing as part of the command.
I even put a carrot (^) in front of it, LOOKS fine in the prompt window, but mysql still gets that ^ passed to it, and doesn't execute the command.
Any suggestions? Thanks!
You could try:
type alter1.sql | mysql -f -utest -ppass db