This question already has answers here:
How can I output MySQL query results in CSV format?
(40 answers)
Closed 3 years ago.
I am trying to export a result set into a csv file and load it to mysql.
mysql -e "select * from temp" > '/usr/apps/{path}/some.csv'
The out put file is not importable. It has the query, headers and bunch of unwanted lines. All I want is just the COMMA delimited VALUES in the file, so that I can import it back.
What did I try so far?
Added | sed 's/\t/,/g' - Did not help
Tried OUTFILE but it did not work.
Tried SHOW VARIABLES LIKE "secure_file_priv" which gave null.
OUTFILE will not work for me because I get the error "The MySQL server is running with the --secure-file-priv option so it cannot execute this statement". I cannot edit the variable secure-file-priv. And it has a null value right now.
I get the file output as below image. I used the alias mysql2csv='sed '\''s/\t/","/g;s/^/"/;s/$/"/;s/\n//g'\'''
This page shows you how to export to a CSV using the command line:
https://coderwall.com/p/medjwq/mysql-output-as-csv-on-command-line
From that page:
# add alias to .bashrc
alias mysql2csv='sed '\''s/\t/","/g;s/^/"/;s/$/"/;s/\n//g'\'''
$ mysql <usual args here> -e "SELECT * FROM foo" | mysql2csv > foo.csv
Since you're trying things, why not try something like the example given in the MySQL Reference Manual?
https://dev.mysql.com/doc/refman/8.0/en/select-into.html
Excerpt:
Here is an example that produces a file in the comma-separated values (CSV) format used by many programs:
SELECT a,b,a+b INTO OUTFILE '/tmp/result.txt'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM test_table;
From a shell script, if we are going to have literal SQL in the script, we may need to take care with the single quote and the backslash characters. For example, the \n (backslash n characters) need to be sent to the MySQL server as part of the SQL text. We have to be careful that the backslash character doesn't get swallowed by the shell script.
Related
After trying for a full day, I'm hoping someone here can help me make below script work. I've combined information from multiple threads (example) and websites, but can't get it to work.
What I'm trying to do:
I'm trying to get a MariaDB10 database called 'stock_db' on my Synology NAS to load all *.csv files from a specific folder (where I save downloaded historical prices of stocks) and add these to a table called 'prices'. The files are all equally named "price_history_'isin'.csv".
Below SQL statement works when running it individually from HeidiSQL on my Windows machine:
Working SQL
LOAD DATA LOW_PRIORITY LOCAL INFILE 'D:\\Downloads\\price_history_NL0010366407.csv'
IGNORE INTO TABLE `stock_db`.`prices`
CHARACTER SET utf8
FIELDS TERMINATED BY ';'
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 2 LINES
(#vdate, #vprice)
SET
isin = 'NL0010366407',
date = STR_TO_DATE(#vdate, '%d-%m-%Y'),
price = #vprice
;
The issue
Unfortunately, when I try to batch loading all csv's from a folder on my NAS through below script, I keep getting the same error.
#!/bin/bash
for filename in ./price_history/*.csv; do
echo $filename
isin=${filename:30:12}
echo $isin
/volume1/#appstore/MariaDB10/usr/local/mariadb10/bin/mysql -u root -p \
"LOAD DATA LOW_PRIORITY LOCAL INFILE '$filename'\
IGNORE INTO TABLE 'stock_db.prices'\
CHARACTER SET utf8\
FIELDS TERMINATED BY ';'\
OPTIONALLY ENCLOSED BY '"'"'"'\
ESCAPED BY '"'"'"'\
LINES TERMINATED BY '\r\n'\
IGNORE 2 LINES (#vdate, #vprice)\
SET\
isin = '$isin',\
date = STR_TO_DATE(#vdate, '%d-%m-%Y'),\
price = #vprice;"
done
ERROR 1102 (42000): Incorrect database name
What I've tried
Took the database name out of stock_db.prices and mentioned it separately as [database] outside of the quoted SQL statement - Doesn't work
Changed quotes around 'stock_db.prices' in many different ways - Doesn't work
Separated the SQL into a separate file and referenced it '< stmt.sql' - Complicates things even further and couldn't get it to work at all (although probably preferred)
Considered (or even preferred) using a PREPARE statement, but seems I can't use this in combination with LOAD DATA (reference)
Bonus Question
If someone can help me do this without having to re-enter the user's password or putting the password in the script, this would be really nice bonus!
Update
Got the 'Incorrect Database Error' resolved by adding '-e' option
Now I have a new error on the csv files:
ERROR 13 "Permission Denied"
While the folder and files are full access for everyone.
Anyone any thoughts to this?
Thanks a lot!
Try to set database using -D option: change the first line to
/volume1/#appstore/MariaDB10/usr/local/mariadb10/bin/mysql -D stock_db -u root -p \ ...
You may have an error in this line IGNORE INTO TABLE 'stock_db.prices'\ - try to remove the single quotes.
Create file .my.cnf in your user's home directory and put the following information into it:
[client]
password="my password"
Info about option files.
'stock_db.prices'
Incorrect quoting. This will work since neither are keywords:
stock_db.prices
This will also work:
`stock_db`.`prices`
Note that the db name and the table name are quoted separately, using backtics.
I can't predict what will happen with this nightmare:
'"'"'"'
This question already has answers here:
Exporting results of a Mysql query to excel?
(9 answers)
Closed 8 years ago.
How do you save output of a MySQL query to a MS Excel sheet?
Even if it's only possible to store the data in a .txt file, it will be okay.
From Save MySQL query results into a text or CSV file:
MySQL provides an easy mechanism for writing the results of a select
statement into a text file on the server. Using extended options of
the INTO OUTFILE nomenclature, it is possible to create a comma
separated value (CSV) which can be imported into a spreadsheet
application such as OpenOffice or Excel or any other application which
accepts data in CSV format.
Given a query such as
SELECT order_id,product_name,qty FROM orders
which returns three columns of data, the results can be placed into
the file /tmp/orders.txt using the query:
SELECT order_id,product_name,qty FROM orders
INTO OUTFILE '/tmp/orders.txt'
This will create a tab-separated file, each row on its own line. To
alter this behavior, it is possible to add modifiers to the query:
SELECT order_id,product_name,qty FROM orders
INTO OUTFILE '/tmp/orders.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
In this example, each field will be enclosed in double quotes, the
fields will be separated by commas, and each row will be output on a
new line separated by a newline (\n). Sample output of this command
would look like:
"1","Tech-Recipes sock puppet","14.95" "2","Tech-Recipes chef's hat","18.95"
Keep in mind that the output file must not already exist and that the
user MySQL is running as has write permissions to the directory MySQL
is attempting to write the file to.
Syntax
SELECT Your_Column_Name
FROM Your_Table_Name
INTO OUTFILE 'Filename.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
Or you could try to grab the output via the client:
You could try executing the query from the your local client and
redirect the output to a local file destination:
mysql -user -pass -e "select cols from table where cols not null" > /tmp/output
Hint: If you don't specify an absoulte path but use something like INTO OUTFILE 'output.csv' or INTO OUTFILE './output.csv', it will store the output file to the directory specified by show variables like 'datadir';.
You can write following codes to achieve this task:
SELECT ... FROM ... WHERE ...
INTO OUTFILE 'textfile.csv'
FIELDS TERMINATED BY '|'
It export the result to CSV and then export it to excel sheet.
I'm trying to import a large csv file wiht 27797 rows into MySQL. Here is my code:
load data local infile 'foo.csv' into table bar fields terminated by ',' enclosed by '"' lines terminated by '\n' ignore 1 lines;
It works fine. However, some rows of this file containing backslashes (\), for example:
"40395383771234304","40393156566585344","84996340","","","2011-02-23 12:59:44 +0000","引力波宇宙广播系统零号控制站","#woiu 太好了"
"40395151830421504","40392270645563392","23063222","","","2011-02-23 12:58:49 +0000","引力波宇宙广播系统零号控制站","#wx0 确切地讲安全电压是\""不高于36V\""而不是\""36V\"", 呵呵. 话说要如何才能测它的电压呢?"
"40391869477158912","40390512645124096","23063222","","","2011-02-23 12:45:46 +0000","引力波宇宙广播系统零号控制站","#wx0 这是别人的测量结果, 我没验证过. 不过麻麻的感觉的确是存在的, 而且用适配器充电时麻感比用电脑的前置USB接口充电高"
"15637769883","15637418359","35192559","","","2010-06-07 15:44:15 +0000","强互作用力宇宙探测器","#Hc95 那就不是DOS程序啦，只是个命令行程序，就像Android里的adb.exe。$ adb push d:\hc95.tar.gz /tmp/ $ adb pull /system/hc95/eyes d:\re\"
After importing, lines with backslashes will be broken.
How could I fix it? Should I use sed or awk to substitute all \ with \ (within 27797 rows...)? Or this can be fixed by just modifying the SQL query?
This is abit more of a discussion than a direct answer. Do you need the double quotes in the middle of the values in the final data (in the DB)? The fact that you have a large amount of data to munge doesn't present any problems at all.
The "" thing is what Oracle does for quotes inside strings. I think whatever built that file attempted to escape the quote sequence. This is the string manual for MySQL. Either of these is valid::
select "hel""lo", "\"hello";
I would tend to do the editing separately to the import, so it easier/faster to see if things worked. If your text file is less than 10MB, it shouldn't take more than a minute to update it via sed.
sed -e 's/\\//' foo.csv
From your comments, you can set the escape char to be something other than '\'.
ESCAPED BY 'char'
This means the loader should verbatim add the values. If it gets too complicated, if you base64() the data before you insert it, this will stop any tools from breaking the UTf8 sequences.
What I did in a similar situation was to create a java string first in a test application. Then compile the test class and fix any errors that I found.
For example:
`String me= "LOAD DATA LOCAL INFILE 'X:/access.log/' REPLACE INTO TABLE `logrecords"+"`\n"+
"FIELDS TERMINATED BY \'|\'\n"+
"ENCLOSED BY \'\"\'\n"+
"ESCAPED BY \'\\\\\'\n"+
"LINES TERMINATED BY \'\\r\\n\'(\n"+
"`startDate` ,\n"+
"`IP` ,\n"+
"`request` ,\n"+
"`threshold` ,\n"+
"`useragent`\n"+
")";
System.out.println("" +me);
enter code here
This is a simple .bat file which currently 'works'; I'm looking to avoid having the field name as the first line in the text file.
C:\xampp\mysql\bin\mysql -u sample -pnotpass --database test -e "SELECT url FROM single WHERE fold = 'bb' AND dir= 'test_folder' ; " > test123.txt
Obviously, it's not a "Windows related question"; Is there a way to ask mysql to only print the results and skip the field name?
Thanks
There's a "column-names" paramater which defaults to true. Just set it to false.
C:\xampp\mysql\bin\mysql -u sample -pnotpass --database test --column-names=false -e "SELECT url FROM single WHERE fold = 'bb' AND dir= 'test_folder' ; " > test123.txt
It depends on the precise format you require, but I'm guessing that using SELECT INTO OUTFILE would probably be a step in the right direction. It would also remove the need to redirect the content to a file, although you'll need to remove that file once you've finished with it (or at the start of the batch file) otherwise MySQL will spit tacks the next time it tries to create it and discovers it already exists.
For example, to create a CSV style file you could use:
SELECT url INTO OUTFILE "\wherever\test123.txt"
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY "\n"
FROM single WHERE fold = 'bb' AND dir= 'test_folder' ;
Incidentally, the user in question must have the FILE privilege for INTO OUTFILE to work.
How do I write the results from a mysql query to file? I just need something quick. Output can be CSV, XML, HTML, etc.
SELECT a,b,a+b
FROM test_table
INTO OUTFILE '/tmp/result.txt'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(the docs show INTO OUTFILE up in the SELECT .. portion which may work as well, but I've never tried it that way)
http://dev.mysql.com/doc/refman/5.0/en/select.html
INTO OUTFILE creates a file on the server; if you are on a client and want it there, do:
mysql -u you -p -e "SELECT ..." > file_name
if you have phpMyAdmin installed, it is a nobrainer: Run the query (haven't got a copy loaded, so I can't tell you the details, but it really is easy) and check neer bottom for export options. CSV will be listed, but I think you can also have SQL if you like :)
phpMyAdmin will give CSV in Excels dialect, which is probably what you want...
You can use MySQL Query Browser to run the query and then just go to File -> Export Resultset and choose the output format. The options are CSV, HTML, XML, Excel and PLIST.