I need to get the result of mysql query execution results saved into a txt file.
The file contains over 600000 lines of insert statements.
The command that I used is:
mysql -h10.100.109.123 -f -uUserName -pPassword
--database=databaseName < AFC.sql > abc.txt
Abc.txt gets created but there is no data in it. From console, I can see there are many errors. -f is used so that it wont stop on error.
Have gone through and tried out many suggestions of similar question. But none of them worked.
MySql version 5.7
How do I pass a parameter (numeric) from the command line to MySQL query file?
mysql --host=<hostname> --user=<username> -p --port=#### < Query.sql > Output.csv
You probably need to transform your Query.sql file, then pass it to mysql. The mysql program itself doesn't have any parameter substitution capability.
For example, in linux or freebsd.
cat 'SELECT #numericParameter := 42;' >/tmp/param$$
cat /tmp/param$$ Query.sql | mysql --host=<hostname> --user=<username> -p --port=#### > Output.csv
rm /tmp/param$$
The first line of this shell script writes one line of SQL, setting a parameter, into a temp file.
The second line sticks that one line at the beginning of your SQL file and pipes the result to the mysql client program.
The third line deletes the temp file.
This leaves your SQL file unmodified when it's done. That's probably what you want.
i am trying to rewrite a script that is written in c-shell script to that uses sql plus command to get information from an oracle database but i am replacing it with mysql and i would like to replace all sqlplus syntax with mysql syntax. I am asking all the c-shell gurus to explain to me what this command means
set SQLPLUS=${ORACLE_HOME}/bin/sqlplus
set REPORT=${MYBD_HOME}/Scripts/report.sql
so somewhere along the line i invoke the sql plus command using the follwing
${SQLPLUS} ${MYDBUSER} # &{REPORT}
i am able to say i undertand what the right hand values mean ({ORACLE_HOME}/bin/sqlplus) is the path to where my sqplus command is located and thus i need it to invoke the command and the {REPORT=$(MYBD_HOME}/Scripts.report.sql) is the path where my sql script that is to be ran by invoking the sqplus command resides correct?
what i dont understand is what the set command is initializing this to. is SQLPLUS a variable so i dont have to type the path when i try to put it in my .csh script?
If so then all i need to do to run this script on a mysql database is simply set the SQLPLUS(problably change it to MYSQL) to point to the path where my msql exec is right
set MYSQL=${MYSQL_HOME}/bin/mysql
then just invoke mysql and run the sql statement
${MYSQL}${MYDBUSER}#${REPORT}
is this what i need to do ro tun the same .tsch script to get data from a mysql table?
You'll need something like this:
${MYSQL} -u $username -p$password -D $database < ${REPORT}
(The username and password are passed in differently to the mysql executable than they are passed to SQLPlus. You'll need to parse out the username and the password from ${MYDBUSER}. Likely, that contains a string such as "scott/tiger". The equivalent on the mysql command line client would be "-u scott -ptiger -D scott".
That # (at sign) is a SQLPlus thing; it tells SQLPLus to read input from the specified filename. The equivalent in mysql would be the source command, e.g.
${MYSQL} -u $username -p$password <_EOD!
use $database
source ${REPORT}
_EOD!
Also, your report.sql file likely includes spool and other SQLPLus specific commands. The mysql command line client is NOT ANYWHERE near as powerful a reporting tool as SQLPlus is.
Addendum:
Q: what exactly does the spool do?
The SQLPlus spool command directs output to a file. It's frequently used to create a log file from a SQLPLus session, and is also useful for creating report files.
set trimspool on
spool /tmp/file.lis
select 'foo' as foo from dual;
spool off
Q: Why can't i set the user name and passowrd to a variable and use that?
You could set a variable, the end result of the command line sent to the OS would be the same.
set MYDBUSER="-u username -ppassword -D database"
${MYSQL} ${MYDBUSER} <${REPORT}
Q:Seems like mysql is more verbose than sqlplus.
The mysql command line client takes unix-style options. These are equivalent:
mysql -u myusername -pmypassword -D mydatabase
mysql --user=myusername --password=mypassword --database=mydatabase
A mysqldump command like the following:
mysqldump -u<username> -p<password> -h<remote_db_host> -T<target_directory> <db_name> --fields-terminated-by=,
will write out two files for each table (one is the schema, the other is CSV table data). To get CSV output you must specify a target directory (with -T). When -T is passed to mysqldump, it writes the data to the filesystem of the server where mysqld is running - NOT the system where the command is issued.
Is there an easy way to dump CSV files from a remote system ?
Note: I am familiar with using a simple mysqldump and handling the STDOUT output, but I don't know of a way to get CSV table data that way without doing some substantial parsing. In this case I will use the -X option and dump xml.
mysql -h remote_host -e "SELECT * FROM my_schema.my_table" --batch --silent > my_file.csv
I want to add to codeman's answer. It worked but needed about 30 minutes of tweaking for my needs.
My webserver uses centos 6/cpanel and the flags and sequence which codeman used above did not work for me and I had to rearrange and use different flags, etc.
Also, I used this for a local file dump, its not just useful for remote DBs, because I had too many issues with selinux and mysql user permissions for SELECT INTO OUTFILE commands, etc.
What worked on my Centos+Cpanel Server
mysql -B -s -uUSERNAME -pPASSWORD < query.sql > /path/to/myfile.txt
Caveats
No Column Names
I cant get column names to appear at the top. I tried adding the flag:
--column-names
but it made no difference. I am still stuck on this one. I currently add it to the file after processing.
Selecting a Database
For some reason, I couldn't include the database name in the commandline. I tried with
-D databasename
in the commandline but I kept getting permission errors, so I ended using the following the top of my query.sql:
USE database_name;
On many systems, MySQL runs as a distinct user (such as user "mysql") and your mysqldump will fail if the MySQL user does not have write permissions in the dump directory - it doesn't matter what your own write permissions are in that directory. Changing your directory (at least temporarily) to world-writable (777) will often fix your export problem.
MySQL is awesome! I am currently involved in a major server migration and previously, our small database used to be hosted on the same server as the client. So we used to do this : SELECT * INTO OUTFILE .... LOAD DATA INFILE ....
Now, we moved the database to a different server and SELECT * INTO OUTFILE .... no longer works, understandable - security reasons I believe.
But, interestingly LOAD DATA INFILE .... can be changed to LOAD DATA LOCAL INFILE .... and bam, it works.
I am not complaining nor am I expressing disgust towards MySQL. The alternative to that added 2 lines of extra code and a system call form a .sql script. All I wanted to know is why LOAD DATA LOCAL INFILE works and why is there no such thing as SELECT INTO OUTFILE LOCAL?
I did my homework, couldn't find a direct answer to my questions above. I couldn't find a feature request # MySQL either. If someone can clear that up, that had be awesome!
Is MariaDB capable of handling this problem?
From the manual: The SELECT ... INTO OUTFILE statement is intended primarily to let you very quickly dump a table to a text file on the server machine. If you want to create the resulting file on some client host other than the server host, you cannot use SELECT ... INTO OUTFILE. In that case, you should instead use a command such as mysql -e "SELECT ..." > file_name to generate the file on the client host."
http://dev.mysql.com/doc/refman/5.0/en/select.html
An example:
mysql -h my.db.com -u usrname--password=pass db_name -e 'SELECT foo FROM bar' > /tmp/myfile.txt
You can achieve what you want with the mysql console with the -s (--silent) option passed in.
It's probably a good idea to also pass in the -r (--raw) option so that special characters don't get escaped. You can use this to pipe queries like you're wanting.
mysql -u username -h hostname -p -s -r -e "select concat('this',' ','works')"
EDIT: Also, if you want to remove the column name from your output, just add another -s (mysql -ss -r etc.)
The path you give to LOAD DATA INFILE is for the filesystem on the machine where the server is running, not the machine you connect from. LOAD DATA LOCAL INFILE is for the client's machine, but it requires that the server was started with the right settings, otherwise it's not allowed. You can read all about it here: http://dev.mysql.com/doc/refman/5.0/en/load-data-local.html
As for SELECT INTO OUTFILE I'm not sure why there is not a local version, besides it probably being tricky to do over the connection. You can get the same functionality through the mysqldump tool, but not through sending SQL to the server.
Since I find myself rather regularly looking for this exact problem (in the hopes I missed something before...), I finally decided to take the time and write up a small gist to export MySQL queries as CSV files, kinda like https://stackoverflow.com/a/28168869 but based on PHP and with a couple of more options. This was important for my use case, because I need to be able to fine-tune the CSV parameters (delimiter, NULL value handling) AND the files need to be actually valid CSV, so that a simple CONCAT is not sufficient since it doesn't generate valid CSV files if the values contain line breaks or the CSV delimiter.
Caution: Requires PHP to be installed on the server!
(Can be checked via php -v)
"Install" mysql2csv via
wget https://gist.githubusercontent.com/paslandau/37bf787eab1b84fc7ae679d1823cf401/raw/29a48bb0a43f6750858e1ddec054d3552f3cbc45/mysql2csv -O mysql2csv -q && (sha256sum mysql2csv | cmp <(echo "b109535b29733bd596ecc8608e008732e617e97906f119c66dd7cf6ab2865a65 mysql2csv") || (echo "ERROR comparing hash, Found:" ;sha256sum mysql2csv) ) && chmod +x mysql2csv
(download content of the gist, check checksum and make it executable)
Usage example
./mysql2csv --file="/tmp/result.csv" --query='SELECT 1 as foo, 2 as bar;' --user="username" --password="password"
generates file /tmp/result.csv with content
foo,bar
1,2
help for reference
./mysql2csv --help
Helper command to export data for an arbitrary mysql query into a CSV file.
Especially helpful if the use of "SELECT ... INTO OUTFILE" is not an option, e.g.
because the mysql server is running on a remote host.
Usage example:
./mysql2csv --file="/tmp/result.csv" --query='SELECT 1 as foo, 2 as bar;' --user="username" --password="password"
cat /tmp/result.csv
Options:
-q,--query=name [required]
The query string to extract data from mysql.
-h,--host=name
(Default: 127.0.0.1) The hostname of the mysql server.
-D,--database=name
The default database.
-P,--port=name
(Default: 3306) The port of the mysql server.
-u,--user=name
The username to connect to the mysql server.
-p,--password=name
The password to connect to the mysql server.
-F,--file=name
(Default: php://stdout) The filename to export the query result to ('php://stdout' prints to console).
-L,--delimiter=name
(Default: ,) The CSV delimiter.
-C,--enclosure=name
(Default: ") The CSV enclosure (that is used to enclose values that contain special characters).
-E,--escape=name
(Default: \) The CSV escape character.
-N,--null=name
(Default: \N) The value that is used to replace NULL values in the CSV file.
-H,--header=name
(Default: 1) If '0', the resulting CSV file does not contain headers.
--help
Prints the help for this command.
Using mysql CLI with -e option as Waverly360 suggests is a good one, but that might go out of memory and get killed on large results. (Havent find the reason behind it).
If that is the case, and you need all records, my solution is: mysqldump + mysqldump2csv:
wget https://raw.githubusercontent.com/jamesmishra/mysqldump-to-csv/master/mysqldump_to_csv.py
mysqldump -u username -p --host=hostname database table | python mysqldump_to_csv.py > table.csv
Re: SELECT * INTO OUTFILE
Check if MySQL has permissions to write a file to the OUTFILE directory on the server.
Try setting path to /var/lib/mysql-files/filename.csv (MySQL 8). Determine what files directory is yours by typping SHOW VARIABLES LIKE "secure_file_priv"; in mysql client command line.
See answer about here: (...) --secure-file-priv in MySQL answered in 2015 by vhu user