I am migrating a MySQL 5.5 physical host database to a MySQL 5.6 AWS Aurora database. I noticed that when data is written to a file using INTO OUTFILE, 5.5 writes NULL value as '\N' and empty string as ''. However, 5.6 writes both empty string and NULL as ''.
Query
SELECT * FROM $databasename.$tablename INTO OUTFILE $filename CHARACTER SET utf8 FIELDS ESCAPED BY '\\\\' TERMINATED BY $delimiter;
I found official documents about this:
https://dev.mysql.com/doc/refman/5.6/en/load-data.html
With fixed-row format (which is used when FIELDS TERMINATED BY and
FIELDS ENCLOSED BY are both empty), NULL is written as an empty
string. This causes both NULL values and empty strings in the table to
be indistinguishable when written to the file because both are written
as empty strings. If you need to be able to tell the two apart when
reading the file back in, you should not use fixed-row format.
How do I export NULL as '\N'?
How do I export NULL as '\N'?
First of all that's strange and why you want to do that? But if for some reason you want to export it that way then you will have to change your query from select * to using a CASE expression like
select
case when col1 is null then '\\N' else col1 end as col1,
...
from $databasename.$tablename....
As commented you can as well use IFNULL() function or COALESCE() function for the same purpose.
Related
Data:
1|\N|"First\Line"
2|\N|"Second\Line"
3|100|\N
\N represents NULL in MYSQL & MariaDB.
I'm trying to load above data using LOAD DATA LOCAL INFILE method into a table named ID_OPR.
Table structure:
CREATE TABLE ID_OPR (
idnt decimal(4),
age decimal(3),
comment varchar(100)
);
My code looks like below:
LOAD DATA LOCAL INFILE <DATA FILE LOCATION> INTO TABLE <TABLE_NAME> FIELDS TERMINATED BY '|' ESCAPED BY '' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';
Problem with this code is it aborts with error Incorrect decimal value: '\\N' For column <Column name>.
Question:
How to load this data with NULL values in second decimal column and also without loosing \(Backslash) from third string column?
I'm trying this is MariaDB which is similar to Mysql in most case.
Update:
The error i have mentioned appears like a warning and the data is actually getting loaded into table. But the catch here is with the text data.
For example: Incase of the third record above it is being loaded as \N itself into string column. But i want it to be NULL.
Is there any way to make the software to recognize this null value? Something like decode in oracle?
You can't have it both ways - either \ is an escape character or it is not. From MySQL docs:
If the FIELDS ESCAPED BY character is empty, no characters are escaped and NULL is output as NULL, not \N. It is probably not a good idea to specify an empty escape character, particularly if field values in your data contain any of the characters in the list just given.
So, I'd suggest a consistently formatted input file, however that was generated:
use \\ if you want to keep the backslash in the strings
make \ an escape character in your load command
OR
make strings always, not optionally, enclosed in quotes
leave escape character empty, as is
use NULL for nulls, not \N
BTW, this also explains the warnings you were experiencing loading \N in your decimal field.
Deal with nulls with blanks. that should fix it.
1||"First\Line"
2||"Second\Line"
3|100|
Thats how nulls are handled on CSVs and TSVs. And don't expect decimal datatype to go null as it stays 0, use int or bigint instead if needed. You should forget about "ESCAPED BY"; as long as string data is enclosed by "" that deals with the escaping problem.
we need three text file & 1 batch file for Load Data:
Suppose your file location 'D:\loaddata'
Your text file 'D:\loaddata\abc.txt'
1. D:\loaddata\abc.bad -- empty
2. D:\loaddata\abc.log -- empty
3. D:\loaddata\abc.ctl
a. Write Code Below for no separator
OPTIONS ( SKIP=1, DIRECT=TRUE, ERRORS=10000000, ROWS=5000000)
load data
infile 'D:\loaddata\abc.txt'
TRUNCATE
into table Your_table
(
a_column POSITION (1:7) char,
b_column POSITION (8:10) char,
c_column POSITION (11:12) char,
d_column POSITION (13:13) char,
f_column POSITION (14:20) char
)
b. Write Code Below for coma separator
OPTIONS ( SKIP=1, DIRECT=TRUE, ERRORS=10000000, ROWS=5000000)
load data
infile 'D:\loaddata\abc.txt'
TRUNCATE
into table Your_table
FIELDS TERMINATED BY ","
TRAILING NULLCOLS
(a_column,
b_column,
c_column,
d_column,
e_column,
f_column
)
4.D:\loaddata\abc.bat "Write Code Below"
sqlldr db_user/db_passward#your_tns control=D:\loaddata\abc.ctl log=D:\loaddata\abc.log
After double click "D:\loaddata\abc.bat" file you data will be load desire oracle table. if anything wrong check you "D:\loaddata\abc.bad" and "D:\loaddata\abc.log" file
I'm building an AWS pipeline to insert CSV files from S3 to an RDS MySQL DB. The problem I'm facing is that when it attempts to load the file, it treats blanks as empty strings instead of NULLs. For example, Line 1 of the CSV is:
"3","John","Doe",""
Where the value is an integer in the MySQL table, and of course the error in the pipeline is:
Incorrect integer value: '' for column 'col4' at row 1
I was researching the jdbc MySQL paramaters to modify the connection string:
jdbc:mysql://my-rds-endpoint:3306/my_db_name?
jdbcCompliantTruncation=false
jdbcCompliantTruncationis is just an example, is there any of these parameters that can help me insert those blanks as nulls?
Thanks!
EDIT:
A little context, the CSV files are UNLOADS from redshift, so the blanks are originally NULLs when I put them in S3.
the csv files are UNLOADS from redshift
Then look at the documentation for the Redshift UNLOAD command and add the NULL AS option. For example:
NULL AS 'NULL'
use null as '\N' converts blank to null
unload ('SELECT * FROM table')
to 's3://path' credentials
'aws_access_key_id=sdfsdhgfdsjfhgdsjfhgdsjfh;aws_secret_access_key=dsjfhsdjkfhsdjfksdhjkfsdhfjkdshfs'
delimiter '|' null as '\\N' ;
I resolve this issue using the NULLIF function:
insert into table values (NULLIF(?,''),NULLIF(?,''),NULLIF(?,''),NULLIF(?,''))
Have to move a table from MS SQL Server to MySQL (~ 8M rows with 8 coloumns). One of the coloumns (DECIMAL Type) is exported as empty string with "bcp" export to a csv file. When I'm using this csv file to load data into MySQL table, it fails saying "Incorrect decimal value".
Looking for possible work arounds or suggestions.
I would create a view in MS SQL which converts the decimal column to a varchar column:
CREATE VIEW MySQLExport AS
SELECT [...]
COALESCE(CAST(DecimalColumn AS VARCHAR(50)),'') AS DecimalColumn
FROM SourceTable;
Then, import into a staging table in MySQL, and use a CASE statement for the final INSERT:
INSERT INTO DestinationTable ([...])
SELECT [...]
CASE DecimalColumn
WHEN '' THEN NULL
ELSE CAST(DecimalColumn AS DECIMAL(10,5))
END AS DecimalColumn,
[...]
FROM ImportMSSQLStagingTable;
This is safe because the only way the value can be an empty string in the export file is if it's NULL.
Note that I doubt you can cheat by exporting it with COALESCE(CAST(DecimalColumn AS VARCHAR(50)),'\N'), because LOAD INFILE would see that as '\N', which is not the same as \N.
I want to import a CSV file into version 9.2 but the CSV file has double-quote double-quote in the final column position to represent a NULL value:
"2","1001","9","2","0","0","130","","2012-10-22 09:33:07.073000000",""
which is mapped to a column of type Timestamp. postgreSQL doesn't like the "". I've tried to set the NULL option but maybe I'm not doing it correctly? I've tried NULL as '"" and NULL '' and NULL as '' and NULL "" but without success; here's my command:
COPY SCH.DEPTS
FROM 'H:/backups/DEPTS.csv'
WITH (
FORMAT CSV,
DELIMITER ',' ,
NULL '',
HEADER TRUE,
QUOTE '"'
)
but it fails with an error:
ERROR: invalid input syntax for type timestamp: ""
CONTEXT: COPY depts, line 2, column expirydate: ""
P.S. Is there a way to specify the string representation of Booleans to the COPY command? The utility that produced the CSVs (of which there are many) used "false" and "true".
The empty string ("") isn't a valid timestamp, and COPY doesn't appear to offer a FORCE NULL or FORCE EMPTY TO NULL mode; it has the reverse, FORCE NOT NULL, but that won't do what you want.
You probably need to COPY the data into a table with a text field for the timestamp, probably an UNLOGGED or TEMPORARY table, then use an INSERT INTO real_table SELECT col1, col, col3, NULLIF(tscol,'') FROM temp_table;.
COPY should accept true and false as booleans, so you shouldn't have any issues there.
Alternately, read the CSV with a simple Python script and the csv module, and then use psycopg2 to COPY rows into Pg. Or just write new cleaned up CSV out and feed that into COPY. Or use an ETL tool that does data transforms like Pentaho Kettle or Talend.
This still seems to be an issue 5 years later. I ran into this issue today running PostgreSQL 9.6.8. As a workaround before running the COPY command, I use sed to replace all occurrences of "" with null and then add NULL as 'null' to my COPY command i.e.
sed -i 's/""/null/g' myfile.csv
PGPASSWORD=<pwd> psql -h <host> -p <port> -d <db> -U <user>
-c "\copy mytable from myfile.csv WITH CSV DELIMITER ',' QUOTE '\"' ESCAPE '\\' NULL as 'null';"
I'm trying to import a CSV file into a MySQL 5.1 DB using phpMyAdmin. The file includes several date columns which may contain NULL values. According to the manual, NULL should be written as \N. However, after an otherwise successful import, \N appears as 0000-00-00 in the date columns (as opposed to NULL). How do I get NULLs imported?
Options set:
line separator ,
fields enclosed by ",
fields escaped by \,
lines terminated by auto.
phpMyAdmin distinguishes two CSV import formats: CSV and CSV with LOAD DATA. The latter option actually accepts \N as described in the manual.
If columns are enclosed by a character (as it is the case), NULLs may also be imported using the former method (no LOAD DATA) by setting a value to a non-enclosed NULL in the CSV file. This is in accordance with the manual's following statement:
If FIELDS ENCLOSED BY is not empty, a field containing the literal
word NULL as its value is read as a NULL value. This differs from the
word NULL enclosed within FIELDS ENCLOSED BY characters, which is read
as the string 'NULL'.