I'm trying to import a XML file into a MySQL Table. In the XML file there is a timestamp in <CurrentTime> in the following format:
2016-01-26T09:52:19.3420655+01:00
This timstamp should go into the corresponding DATETIME CurrentTime column in my Table. So I did the following
LOAD XML INFILE 'xxx.xml'
INTO TABLE test.events
ROWS IDENTIFIED BY '<Event>'
SET CurrentTime = str_to_date(CurrentTime, '%Y-%m-%dT%H:%i:%s.%f');
But it quits with the error
Error Code: 1292. Incorrect datetime value: '2016-01-25T16:22:24.1840792+01:00' for column 'CurrentTime' at row 1
So it seems it doesn't convert the string at all. Why?
I think that error is thrown when the string value from the file is loaded directly to the column. The error is thrown before you get to the SET clause.
Here's an abbreviated example of how to use user-defined variables to pass the value of a field down to the SET, bypassing the assignment to the column.
Note that the columns _row and account_number are populated directly from the first two fields in the file. The later fields in the file are assigned to user-defined variables (identifiers beginning with #.
The SET clause evaluates the user-defined variables, and assigns the result of the expression to the actual column in the table.
In this example, the "dates" were formatted YYYYMMDD. I used the STR_TO_DATE() function to have that string converted to a proper DATE.
I abbreviated this sample somewhat, but it demonstrates the approach of reading field values into user-defined variables.
CREATE TABLE _import_water
(`_row` INT
,`account_number` VARCHAR(255)
,`total_due` DECIMAL(18,2)
,`end_date` DATE
,`start_date` DATE
,`ccf` DECIMAL(18,4)
)
LOAD DATA LOCAL INFILE '//server/share$/users/me/mydir/myfile.csv'
INTO TABLE _import_water
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(_row
,account_number
,#total_due
,#end_date
,#start_date
,#ccf
)
SET `total_due` = NULLIF(#total_due,'')
, `end_date` = STR_TO_DATE(#end_date,'%Y%m%d')
, `start_date` = STR_TO_DATE(#start_date,'%Y%m%d')
, `ccf` = NULLIF(#ccf,'')
Also, it doesn't look like there's any problem with your STR_TO_DATE, it seems to evaluate just fine.
testing...
SELECT STR_TO_DATE('2016-01-25T16:22:24.1840792+01:00','%Y-%m-%dT%H:%i:%s.%f') AS mydatetime
returns:
mydatetime
--------------------------
2016-01-25 16:22:24.184079
Related
Here is my code:
CREATE TABLE A
(`ID` INT NULL,
`DATE` DATE NULL,
`NUM` INT NULL
);
LOAD DATA LOCAL INFILE "fakepath/file.csv"
INTO TABLE A
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(ID,DATE,NUM)
SET
DATE = str_to_date(#DATE, '%Y%m%d');
The original data in csv file is like this-- 20160101,20160102,20160103 (the date is different). After I execute the code, all the date in the DATE column become one day value such as 2016-01-02 in table A.
Why do this happen? I have other table which used the same code(different column name)
How can I fix it? Thank you!
You have to tell MySQL to load the from-csv data value into a variable first:
LOAD DATA LOCAL INFILE "fakepath/file.csv"
[..snip..]
IGNORE 1 LINES
(ID,DATE,NUM)
^---table field, **NOT** a variable
SET
DATE = str_to_date(#DATE, '%Y%m%d');
^---variable never gets populated
Try
(ID, #DATE, NUM)
^--note this
instead. That'll load the id/num values directly into the table, but puts your date value into the variable, which you can use afterwards in the SET portion of the query.
The fact that you actually get a date value put into the table with a proper date format indicates that somwhere else, in previous code, you did set a #DATE variable, and it's simply being re-used in this query. But since you don't CHANGE that variable's value in this query, you end up using the SAME date value for all records.
I have created this table on MySQL :
create table Apolo(
Date date,
Name varchar(50)
);
I have imported an excel file :
LOAD DATA LOCAL INFILE 'C:/Users/File.csv'
INTO TABLE Apolo
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 10 ROWS
(#Date, Name);
set Date=str_to_date(#Date,'%d/%m/%Y');
and I get the error :
Error Code: 1193. Unknown system variable 'Date'
If I do not put this line :
set Date=str_to_date(#Date,'%d/%m/%Y');
I do not get the error but if I try to use :
select count(*) from Apolo where Date='03/09/2015';
it does not work. So the format is not recognisable.
The date to insert into mysql database should be in the format
YYYY-MM-DD
Example: 2015-02-18
So change the date in csv file to the above specified format and try ..
Check your Date column values in database. By default date format in mysql database is Y-m-d
Do you have values in this format.
Take the semi-colon out of this line
(#Date, Name);
The set should be in the same command. Watch out for the fact that you have a field name that is a reserved word - you may need to escape it.
Objective
I'm trying to load a .csv file called persondata into DB alldata TABLE persondata and then run a query on it for firstname, lastname and dateofbirth (dob). The .csv has only 4 records in it.
Date format in csv is MM/DD/YYYY, the output should be YYYY/MM/DD, YYYY-MM-DD or YYYYMMDD.
SQL
LOAD DATA LOCAL INFILE 'C:/Users/john.smith/Desktop/persondata.csv'
INTO TABLE alldata.persondata
FIELDS TERMINATED BY ','
(firstname, lastname, dob, apptdate, icd9, cpt)
SET dob = str_to_date(#dob, '%c/%e/%Y')
;
SELECT firstname, lastname, dob
FROM alldata.persondata
Problem and Error Message I'm Getting
firstname and lastname return proper values but dob returns null for all 4 records. In the csv file, the first three colums (A, B, C) are firstname, lastname, dob. So same order as in the table persondata.
Error:
4 row(s) affected, 8 warning(s): 1265 Data truncated for column 'dob' at row 1 1411 Incorrect datetime value: '19850708' for function str_to_date 1265
Help pages I consulted:
(using str_to_date in general)
How to convert csv date format to into mysql db
(using 'SET column = str_to_date...')
MySql load data infile STR_TO_DATE returning blank?
(other)
How to change string date to MySQL date format at time of import of CSV using MySQL's LOAD DATA LOCAL INFILE
Cannot transform mm/dd/yyyy in excel to csv yyyymmdd date format using SSIS
MySQL str_to_date produces NULL despite valid formatting
Additional Information:
I experimented with this query and a bunch of variations of it but no luck:
SET dob = date_format(str_to_date(#dob, '%c/%e/%Y'), '%Y/%c/%e')
I'm not seeing a huge amount of consensus on how to write this. Some people specify the output format of %Y/%d/%m and some don't. But isn't that the only date format that mysql supports? This makes me think I shouldn't have to write it. Not sure if this is even related. I've seen a few syntaxes of the entire thing. I've read through all the support pages and I think that I understand the 'SET' command.
*this is my first post on stackoverflow so please let me know if should present anything differently
You're using "dob" where you should be using "#dob" in your column list... line 4 should be:
(firstname, lastname, #dob, apptdate, icd9, cpt)
^^^^
This is because mysql reads the date into a VARIABLE (indicated by the #), and then the later SET command manipulates the variable to match the actual column (which is identified correctly dob without the #).
Also, I think you're using slashes in your string format when the date doesn't appear to have slashes in it. That is, the error says the date is "19850708" (July 8th, 1985). I believe you want:
%Y%m%d
That is, change line 5 to:
SET dob = str_to_date(#dob, '%Y%m%d')
%m and %d are the 2-digit month and day, which you need since you have '07' instead of just '7' for July, for example. After your comments I'm not sure which date format is correct, but certainly the #dob vs dob variable issue is real. You may want to replace every instance of #dob with something like #original_dob just to alleviate future confusion.
See this page: http://dev.mysql.com/doc/refman/5.1/en/date-and-time-functions.html#function_str-to-date
I've loaded some date from file to table and now i want to convert the string with date to a datetime format.
The string i 'datestring' column looks like this '12-16-2010 01:48:28', and if i run this query:
select STR_TO_DATE('12-16-2010 01:48:28', '%c-%e-%Y %T')
It returns proper datetime: 2010-12-16 01:48:28
But when i try to run this:
update database.`temptable`
SET datetimefile = (SELECT STR_TO_DATE(datestring, '%c-%e-%Y %T'))
I get those kind of errors:
Incorrect datetime value: ''12-16-2010 01:48:28'' for function str_to_date
Any ideas?
Take a close look at the error message:
Incorrect datetime value: ''12-16-2010 01:48:28''
^^ 2 single quotes ^^
Compare this to the normal error message:
mysql> SELECT STR_TO_DATE('foo', '%c-%e-%Y %T');
+-----------------------------------+
| STR_TO_DATE('foo', '%c-%e-%Y %T') |
+-----------------------------------+
| NULL |
+-----------------------------------+
1 row in set, 1 warning (0.00 sec)
mysql> show warnings;
+---------+------+----------------------------------------------------------+
| Level | Code | Message |
+---------+------+----------------------------------------------------------+
| Warning | 1411 | Incorrect datetime value: 'foo' for function str_to_date |
+---------+------+----------------------------------------------------------+
1 row in set (0.00 sec) ^ ^ just 1 single quote
Normally, the error message has a single set of single quotes. Yours has a double set, suggesting that you actually have a set of single quotes stored in your column data.
If this is the case, you can work around this by removing them where they exist:
SET datetimefile = (SELECT STR_TO_DATE(REPLACE(datestring,"'",''), '%c-%e-%Y %T'))
Using REPLACE() like this still would work even if not all of the rows contain the spurious quotes, since replace passes through the input value unchanged if the 'from_str' (2nd arg) doesn't occur.
From PHP SQL Request Correct:
LOAD DATA INFILE '.$filename."' INTO TABLE tablename (#var_DTime, `Product`, `Source`, `Cost`) SET `DTime` = str_to_date(#var_DTime,'%Y-%m-%dT%H:%i:%s')
Do not use: " " - 2010-12-31 01:48:28; - Don't work
Use "T" - 2010-12-31T01:48:28; - Work
I had a similar problem,
I wanted to use order by date syntax, but since my dates were in text format it returned the table unsorted.
I tried using
Alter table
Alter column `column name` date
but it gave the me same error you've got.
the only solution that I found by trial and error was to change the format of the date in the CSV file. I changed it to YYYY-MM-DD, then used "Alter table , alter column column name" to change the data type in the column
This error happened when you are trying to add a non-DateTime value inside a DateTime column.
For example, the DateTime format is YYYY-MM-DD HH:MM:SS. So in any case, if you try to add a date with any format other than the original format, it will throw this error.
In my case, I was using LOAD DATA INFILE to load a CSV file, to a specific table. One of the columns in the CSV file was in another date format, so I was supposed to change the format using str_to_date function using the below syntax:
LOAD DATA INFILE 'x.csv'
INTO TABLE orders FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(
timimg,
order_id,
order_details
)
set timing= STR_TO_DATE(lower(#timing), '%d-%b-%Y %H:%i:%s');
The above syntax throws the same error (Incorrect DateTime value for function str_to_date), So what was the issue?.
It was just adding the # for the timing column. # character means that you are telling MySQL to not load the data directly in the table for that column, however, make some changes to the value from CSV before inserting in the table.
So by adding the character # in the above query, It fixes the issue. The correct syntax is:
LOAD DATA INFILE 'x.csv'
INTO TABLE orders FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(
#timimg,
order_id,
order_details
)
set timing= STR_TO_DATE(lower(#timing), '%d-%b-%Y %H:%i:%s');
... SET datetimefile = STR_TO_DATE(datestring, '%c-%e-%Y-%T')
Note the lack of select around the str_to_date call. That select had no table reference, so the query failed with "unknown field datestring". That failure bubbled upwards and killed the entire overall query.
I'm trying to convert timestamps on the fly when importing a csv file into mysql from string to datetime data type. But I am getting a #1411 - Incorrect datetime value: '2007-03-30 16:01:15' for function str_to_date error.
The SQL:
load data infile 'C:/ProgramData/MySQL/MySQL Server 5.5/data/testfile.csv'
into table test
fields terminated by ','
lines terminated by '\n'
(date, col1,col2,col3,col4)
SET
date = str_to_date(date,'%Y.%m.%d %H:%i:%s.%f');
All rows in the .csv are formated like this:
2007.03.30 16:01:15.901,117.53,117.55,35600000,43700000
I've applied
SELECT str_to_date(date,'%Y.%m.%d %H:%i:%s.%f') FROM test
to sample data that was already stored in mysql, it did work.
The target row date is set to DATETIME.
You need to go via a user variable. As the manual says:
The column list can contain either column names or user variables. With user variables, the SET clause enables you to perform transformations on their values before assigning the result to columns.
User variables in the SET clause can be used in several ways. The following example uses the first input column directly for the value of t1.column1, and assigns the second input column to a user variable that is subjected to a division operation before being used for the value of t1.column2:
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(column1, #var1)
SET column2 = #var1/100;
In your case:
LOAD DATA INFILE 'C:/ProgramData/MySQL/MySQL Server 5.5/data/testfile.csv'
INTO TABLE test
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(#date, col1, col2, col3, col4)
SET date = STR_TO_DATE(#date, '%Y.%m.%d %H:%i:%s.%f');