Extra backslash \ when SELECT ... INTO OUTFILE ... in MySQL - mysql

So I'm trying to export a MySQL table into CSV. I'm using this query:
SELECT * FROM business WHERE id > 0 AND id <= 20000 INTO OUTFILE "business.csv"
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY "\n";
That output something like this:
http://postimage.org/image/2ghyenh5w/full/
The problem with this is that there's always an extra backslash \ where there's newline such as in the address field.
However, CSV exported from phpMyAdmin doesn't have it:
http://postimage.org/image/2gi026tno/full/
Any way to make the SELECT ... OUTFILE ... do the same?
The table I'm dealing with has 20 million records, phpMyAdmin can only handle about 500,000 records for every export action - or it will go blank or mysql server gone away, etc.

It looks like it's impossible for a MySQL export to correctly export both Newlines and Quotes.
When exporting, MySQL will automatically escape both
Field delimiters, and
Line delimiters
By default, the escape character is a backslash. You can override this by adding ESCAPED BY '' to your query.
Unfortunately, in a "normal" (Excel-compatible) CSV file, you probably want different encodings for newlines and quotes. Specifically, you want newlines to be unescaped, and quotes to be doubled.
E.g. If a value contains a newline like so:
This is line 1
And this is "Line 2", which contains quotes
it should become
"This is line 1
And this is ""Line 2"", which contains quotes"
The solution I found was to pre-escape the quotes, and add ESCAPED BY '' (an empty string) to my query.
SELECT REPLACE(field1, '"', '""'),
REPLACE(field2, '"', '""'),
...
FROM ...
WHERE ...
INTO OUTFILE '/someFile.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY ''
LINES TERMINATED BY '\n'

Try this:
SELECT * FROM business WHERE id > 0 AND id <= 20000 INTO OUTFILE "business.csv"
fields terminated by ',' OPTIONALLY ENCLOSED BY '"' escaped by '"'
LINES TERMINATED BY '\n';
I think the issue is that MySQL is trying to escape newline ('\n') in your text fields because it's your line terminator.
FIELDS ESCAPED BY controls how to write special characters. If the FIELDS ESCAPED BY character is not empty, it is used as a prefix that precedes following characters on output:
The FIELDS ESCAPED BY character
The FIELDS [OPTIONALLY] ENCLOSED BY character
The first character of the FIELDS TERMINATED BY and LINES TERMINATED BY values
ASCII NUL (the zero-valued byte; what is actually written following the escape character is ASCII “0”, not a zero-valued byte)
(MySQL)
I don't really understand why it's doing what it's doing in your case, but I was able to get something like that on my Mac and the query above seemed to fix the output in my case.
Hope that helps!

I had the same problem, and I found out (after importing the csv file into a spreadsheet) that there were line breaks in some varchar fields in the MySQL table. After deleting the line breaks, the export worked correctly.

Try this:
SELECT * FROM business WHERE id > 0 AND id <= 20000 INTO OUTFILE "business.csv"
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\'
LINES TERMINATED BY '\n';
I realised using escaped by '\' makes removes the backslash on exported results.

Got an answer here https://bugs.mysql.com/bug.php?id=46434
Main points are:
1. INTO OUTFILE is intended to produce results ready to load by LOAD DATA
2. By default ESCAPED BY is '\'
3. To disable escaping, use ESCAPED BY ''

I solved this by specifying \r\n as the line terminator, rather than \n:
SELECT * FROM business WHERE id > 0 AND id <= 20000
INTO OUTFILE "business.csv"
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '\'
LINES TERMINATED BY '\r\n';
Each row is now separated by \r\n, but any newlines inside your data will be left unescaped – assuming that the line separators present therein are all \n, not \r\n.
Surprisingly, this worked fine on Linux for my purposes – importing using League\Csv (PHP). I'm guessing that whatever software will be importing your generated CSVs has to be smart enough to differentiate between \n and \r\n for line breaks.

SELECT * FROM business WHERE id > 0 AND id <= 20000 INTO OUTFILE "business.csv"fields terminated by ','
OPTIONALLY ENCLOSED BY '"' ESCAPED BY '' LINES TERMINATED BY '\n';
First, do not put '"" as escape, this will change your content.
Second, if you are using this query on a cli as bellow you need to also remote an extra '\n' add row with multiple lines.
mysql -e "SELECT * FROM business WHERE id > 0 AND id <= 20000 INTO OUTFILE "business.csv"
fields terminated by ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY ''
LINES TERMINATED BY '\n';"

I experienced similar problem.
To prevent extra backlash \ where there are newlines in field values, I added ESCAPED BY '\'.
To prevent MySQL terminating fields where newlines exist within a field value, I replaced newline \n with space ' '.
SELECT
REPLACE(field1, '\n', ' '),
REPLACE(field2, '\n', ' '),
REPLACE(fieldN, '\n', ' ')
FROM business
WHERE id > 0 AND id <= 20000 INTO OUTFILE "business.csv"
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\' LINES TERMINATED BY '\n';

Related

How to export a CSV file from the MySQL command line

I have a MySQL database containing a single table which is used as a temporary storage point for manipulating and querying data sets and is periodically deleted and replaced with new data. What I would like to be able to do is export the data from the MySQL command line to use for other purposes. I am using the XAMPP Apache server package with phpMyAdmin on Windows 10.
The issue I am having is the INTO OUTFILE syntax I am using returns an error relating to '\n'. Below is an example of the syntax:
SELECT *
FROM tablename
WHERE work_complete = 'Yes'
INTO OUTFILE 'C:\file path for the file to be exported to\file_name.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY "''"
ESCAPED BY '\'
LINES TERMINATED BY '\n';
I have spent some time researching this without any luck, I have even tried using
LINES TERMINATED BY '\r\n'
but the error remained the same
ERROR: Unknown command '\''.
-> LINES TERMINATED BY '\r\n';
If anyone could provide any tips that would be greatly appreciated.
Use this
SELECT *
FROM tablename
#WHERE work_complete = 'Yes'
INTO OUTFILE 'C:\file path for the file to be exported to\file_name.csv'
FIELDS ENCLOSED BY '"'
TERMINATED BY ';'
ESCAPED BY '"'
LINES TERMINATED BY '\r\n';
I used some other syntax as you and deleted the OPTIONALLY which mysql doesn't like at that place
The problem is not in LINES TERMINATED BY but in ESCAPED BY.
This:
ESCAPED BY '\'
Is invalid syntax, because the backslash is interpreted as an escape character for the following quote. A decent text editor should let you see that.
You need to escape the backslash, like so:
ESCAPED BY '\\'
Alternatively, you can also use '\b':
ESCAPED BY '\b'
Another probem is that OPTIONALLY ENCLOSED accepts only a single character, while you are giving it two single quotes.
In your query:
SELECT * FROM tablename WHERE work_complete = 'Yes'
INTO OUTFILE 'C:\file path for the file to be exported to\file_name.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY "'" ESCAPED BY '\\'
LINES TERMINATED BY '\n'

How to export data to csv in mysql without the header?

MySQL has the functionality to export the data to a CSV file with following statement
SELECT
*
FROM
person
INTO OUTFILE 'person.csv'
FIELDS ENCLOSED BY '"'
TERMINATED BY ','
ESCAPED BY '"'
LINES TERMINATED BY '\r\n';
But by default it will has the column name as the CSV header, so how to remove the header in this statement?
I reviewed the mysql reference but seems these is no such info.
2 options are provided in this post : How can I suppress column header output for a single SQL statement?)
1 : invoke mysql with -N flag will skip all column headers
2 : fake it
select column1 as '', column2 as '' from some_table;
After complete the csv file, seems for the huge records, the big csv file has no header, so if you want to add the header, following:
Several users asked about including headers, i.e., column names or variable names, in the "INTO OUTFILE" syntax.
One approach is to use the "--column-names" option in the mysql invocation:
mysql --column-names -e 'SELECT * FROM mysql.user' > test.dat
(This creates a tab-delimited file test.dat with column names in the first row followed by the query results.)
SELECT
*
FROM
person
INTO OUTFILE 'person.csv'
FIELDS ENCLOSED BY '"'
TERMINATED BY ','
ESCAPED BY '"'
LINES TERMINATED BY '\r\n';
change FIELDS ENCLOSED BY '"' line
with OPTIONALLY ENCLOSED BY '"'

Importing a CSV in MySQL. Only every second line is imported

i want to import a csv table into mysql with this command:
LOAD DATA LOCAL INFILE 'C:/Users/user/Desktop/SA01505.csv' INTO TABLE hks.orc CHARACTER SET 'utf8' COLUMNS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES (#var1,#var2,#var3,`TI 41-40`,`TI 41-41`,`TI 41-42`,`TI42-40`,`TI 42-41`,`TI 42-42`,`TI 42-43`,`TI 42-43.2`,`TI 42-44`,`TI 42-45`,`TI 42-46`,`TI 42-47`,`PI 42-71`,`PI 42-72`,`PI 42-73`,`PI 42-75`,`FI 42-90`,`TI 43-40`,`TI 43-41`,`TI 45-40`,`TI 45-41`,`TI 47-41`,`TI 47-42`,`TI 47-43`,`TI 47-44`,`LI 42-61`,`SI 44-81`,`QI 46-22`,`UV 41-10_Soll`,`UV 41-10_Ist`,`UV43-10_Soll`,`UV43-10_Ist`,`Tc2`,`Tc3`,`Tc4`)
SET Datum = STR_TO_DATE(CONCAT(#var2,#var3),'%Y/%m/%d%k:%i:%s');
This code works, but imports only every second line, although every line is terminated with "\n" (i checked this with an hex-editor)
I'd guess that the problem is here: OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'. This is biting each other.
In the first line imported the last " of the enclosed column escapes the \n of the line terminator. Therefore always two lines are treated as one and you probably don't see the "second line" in the imported data because the data gets truncated.
Remove the ESCAPED BY '"' clause and see if it works. If it works but you need it to properly import your data, you'll have to work on your data.

How to export MySQL table into CSV without newlines in VARCHAR fields? [duplicate]

I am pretty inexperienced in SQL, so there should be a simple solution to my problem:
I am selecting a table into a comma-separated file, and the column of type TEXT has newline characters, so when I try to import my csv into Excel, it creates separate rows each piece of text following a newline character.
Here is my query:
SELECT * FROM `db`.`table` INTO OUTFILE 'c:\\result.txt' FIELDS TERMINATED BY ','
ESCAPED BY '\\' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\r\n' ;
and then in Excel I import as a comma separated file which causes issues for column that has text with newline characters.
any help is appreciated!
Just enclose everything in double quotes perhaps.
SELECT * FROM db.table INTO OUTFILE 'c:/result.txt' FIELDS TERMINATED BY ',' ESCAPED BY '\\' ENCLOSED BY '"' LINES TERMINATED BY '\r\n';
Novikov is correct but you could also escape the new line characters while exporting.
SELECT REPLACE(`fieldname1`,'\n','\\n'),`fieldname2` FROM db.table INTO OUTFILE 'c:/result.txt' FIELDS TERMINATED BY ',' ESCAPED BY '\\' ENCLOSED BY '"' LINES TERMINATED BY '\r\n';
This will then replace all the new line characters with the text string '\n' This may not be what you want in the output though.
DC

PostgreSQL csv import from a MySQL csv export?

Will it work?
MySQL export:
SELECT * INTO OUTFILE 'C:/data.csv'
FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '\\'
LINES TERMINATED BY '\n'
FROM table;
PostgreSQL:
COPY table FROM 'C:/data.csv' WITH DELIMITER AS '\t' NULL AS '\\N' CSV
There is missing columns for some reason. So I believe there is problem in delimiter. Am I correct, what can I do? I can inspect row with cause error below. But which characters I must look for?
ERROR: missing data for column "Column21"
CONTEXT: COPY table, line 88219: ...
As mentioned by #knitti postgres need to know escape character: ESCAPE '\'
OPTIONALLY ENCLOSED BY '"' is bad format for csv. It's better to force quoting.
Full code:
mysql 5.5.15:
SELECT *
INTO OUTFILE 'C:/data.csv'
FIELDS TERMINATED BY '\t' ENCLOSED BY '"' ESCAPED BY '\\'
LINES TERMINATED BY '\n'
FROM table;
postgres 9.0.3:
COPY table FROM 'C:/data.csv' WITH DELIMITER AS E'\t' NULL AS '\\N' ESCAPE E'\\' CSV
Default quote character in PostgreSQL is double-quote, but the documentation says, you can use any single one-byte-character. Try adding
ESCAPE '\\'