I know similar questions have been posted before, but when I try to follow similar approaches as per the suggestions in the comments, it simply does not help. My query is the following:
LOAD DATA INFILE 'File.txt'
IGNORE
INTO TABLE table_name
FIELDS TERMINATED BY '^~'
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS
(RUN_DATE, PROC_DT, STL_DT, TRD_DT)
SET RUN_DATE = STR_TO_DATE(RUN_DATE, '%d-%b-%y');
The records in the file look something like this:
RUN_DATE^~PROC_DT^~STL_DT^~TRD_DT
21-DEC-20^~23-DEC-20^~23-DEC-20^~21-DEC-20
The dates that get loaded are all populated as '0000-00-00 00:00:00' which I know are the default values when there is a datatype error and IGNORE is used. From what I found online, the issue has to do with the in-file date not being in yyyy-mm-dd format which is the default for mySQL, but the '%d-%b-%y' in the STR_TO_DATE function should help alleviate this issue since
%d: Day of the month as a numeric value (01 to 31) -
%b: Abbreviated month name (Jan to Dec) -
%y: Year as a numeric, 2-digit value
Why is this not helping? I also tried making the months lower case using LOWER() thinking maybe the abbreviated months needed to be all lower case, but this produces the same result. What am I missing here?
To read from the file but store a modified value, you need to use variables:
LOAD DATA INFILE 'File.txt'
IGNORE
INTO TABLE table_name
FIELDS TERMINATED BY '^~'
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS
(#RUN_DATE, #PROC_DT, #STL_DT, #TRD_DT)
SET RUN_DATE = STR_TO_DATE(#RUN_DATE, '%d-%b-%y'),
PROC_DT = STR_TO_DATE(#PROC_DT, '%d-%b-%y'),
STL_DT = STR_TO_DATE(#STL_DT, '%d-%b-%y'),
TRD_DT = STR_TO_DATE(#TRD_DT, '%d-%b-%y');
Related
when using LOAD DATA LOCAL and having a date in the first column then it always stores '0000-00-00 00:00:00' for only the very first row.
When placing a string column as very first column instead everything works fine.
See cases A and B:
A)
this first two rows in csv:
2022-01-31,hello_world
2022-01-31,hello_stackover
lead to this data in the mysql table:
0000-00-00 00:00;00,hello_world
2022-01-31 00:00:00,hello_stackover
B)
this first two rows in csv:
hello_world,2022-01-31
hello_stackover,2022-01-31
lead to this data in the mysql table:
hello_world,2022-01-31 00:00:00
hello_stackover,2022-01-31 00:00:00
This is my sql:
LOAD DATA LOCAL
INFILE 'myfile.csv'
INTO TABLE mydatabase.`mytable`
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\'
LINES TERMINATED BY '\r\n'
(my_date, any_string_column);
Any idea why 0000-00-00 00:00:00 only happens for the very first row and only when it is the very first column? Thanks.
Edit:
8.0.30-0ubuntu0.20.04.2
Your editor is saving the file with a BOM (Byte Order Mark) character at the beginning, which MySQL doesn't recognize. You need to disable this setting.
I have a rather large (over 200,000 records) file that I am inserting into a table with 59 columns. The data contains several DATETIME fields. The input dates are in the form '10/06/2019 10:45:58'. How can I input these fields as DATETIME (or maybe just DATE, since the TIME field is irrelevant for my purposes)?
If I set the various date fields to type DATETIME, the fields come up as 0000-00-00 00:00:00 after loading. That's because the database does not know what to do with the input format.
I see two different approaches, but each has issues:
Preprocessing:
I create a script to detect the date fields using Regex and reformat them to the expected form using something like Perl's DateTime::Format::DBI. There is a risk here since the records include freeform TEXT fields that may contain embedded commas and quotes. Positively identifying the DATE fields is difficult just because of scale.
Post-processing:
Create the table with the date fields as VARCHAR and use the STR_TO_DATE SQL function populate the date columns.
INSERT INTO mytable(DATELastDetected, DATEFirstDetected)
SELECT STR_TO_DATE(LastDetected, '%c/%e/%Y %H:%i'),
STR_TO_DATE(FirstDetected, '%c/%e/%Y %H:%i')
from mytable;
Third Option?
I've wondered whether I might specify the expected format of the input data for that DATE columns in the CREATE TABLE statement, which would render the whole discussion moot. I've seen another question that mentions the use of DATEFORMAT in a CREATE TABLE statement, but I have not found the right syntax to use.
Appreciate any thoughts.
#ben-personick answered it with his comment. Here's what my Load statement looks like:
LOAD DATA INFILE '/opt/mysql/work/report.csv'
INTO TABLE `my_db`.`tbl_reportdata`
CHARACTER SET utf8mb4
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1
LINES (`IP`,
[...]
`OS`,
#FirstDetectedVar,# This field is defined as DATETIME
#LastDetectedVar, # This field is defined as DATETIME
[...]
`Category`)
set
`FirstDetected` = str_to_date(#FirstDetectedVar, '%m/%d/%Y %H:%i'),
`LastDetected` = str_to_date(#LstDetectedVar, '%m/%d/%Y %H:%i');
I figured the answer was out there. Hopefully this working example will help someone else.
I need a way to use the UTC_TIMESTAMP() function in a CSV file.
Currently I'm using the following Syntax;
LOAD DATA INFILE '/path/to/file.csv'
INTO TABLE my_table FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
I'm not quite sure what to do to keep the UTC_TIMESTAMP() from being enclosed in quotes. When entered in the database, I get the following result
"0000-00-00 00:00:00"
This was the only example I could find on stack overflow or on Google for converting a string to a MYSQL value.
MySQL CSV import: datetime value
I solved the problem by following the MySQL documentation on this page.
https://dev.mysql.com/doc/refman/5.7/en/load-data.html
About halfway down the page there is a section that shows you how to create variables for rows and then set table rows equal to native mysql functions or values assigned to those variables(Whichever you choose).
The example in the documentation that I'm referring to looks like this.
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(column1, column2)
SET column3 = CURRENT_TIMESTAMP;
I fixed my problem by restructuring my code like this...
LOAD DATA LOCAL INFILE '/path/to/file.csv'
INTO TABLE veh_icodes FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(id, vcode, year, make, model, body_style, tran_type, engine_cap, drive_train, doors, trim, created_by, updated_by, #created_at, #updated_at, active)
SET created_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP;"
I hope this helps someone. :)
Here is my code:
CREATE TABLE A
(`ID` INT NULL,
`DATE` DATE NULL,
`NUM` INT NULL
);
LOAD DATA LOCAL INFILE "fakepath/file.csv"
INTO TABLE A
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(ID,DATE,NUM)
SET
DATE = str_to_date(#DATE, '%Y%m%d');
The original data in csv file is like this-- 20160101,20160102,20160103 (the date is different). After I execute the code, all the date in the DATE column become one day value such as 2016-01-02 in table A.
Why do this happen? I have other table which used the same code(different column name)
How can I fix it? Thank you!
You have to tell MySQL to load the from-csv data value into a variable first:
LOAD DATA LOCAL INFILE "fakepath/file.csv"
[..snip..]
IGNORE 1 LINES
(ID,DATE,NUM)
^---table field, **NOT** a variable
SET
DATE = str_to_date(#DATE, '%Y%m%d');
^---variable never gets populated
Try
(ID, #DATE, NUM)
^--note this
instead. That'll load the id/num values directly into the table, but puts your date value into the variable, which you can use afterwards in the SET portion of the query.
The fact that you actually get a date value put into the table with a proper date format indicates that somwhere else, in previous code, you did set a #DATE variable, and it's simply being re-used in this query. But since you don't CHANGE that variable's value in this query, you end up using the SAME date value for all records.
I've loaded some date from file to table and now i want to convert the string with date to a datetime format.
The string i 'datestring' column looks like this '12-16-2010 01:48:28', and if i run this query:
select STR_TO_DATE('12-16-2010 01:48:28', '%c-%e-%Y %T')
It returns proper datetime: 2010-12-16 01:48:28
But when i try to run this:
update database.`temptable`
SET datetimefile = (SELECT STR_TO_DATE(datestring, '%c-%e-%Y %T'))
I get those kind of errors:
Incorrect datetime value: ''12-16-2010 01:48:28'' for function str_to_date
Any ideas?
Take a close look at the error message:
Incorrect datetime value: ''12-16-2010 01:48:28''
^^ 2 single quotes ^^
Compare this to the normal error message:
mysql> SELECT STR_TO_DATE('foo', '%c-%e-%Y %T');
+-----------------------------------+
| STR_TO_DATE('foo', '%c-%e-%Y %T') |
+-----------------------------------+
| NULL |
+-----------------------------------+
1 row in set, 1 warning (0.00 sec)
mysql> show warnings;
+---------+------+----------------------------------------------------------+
| Level | Code | Message |
+---------+------+----------------------------------------------------------+
| Warning | 1411 | Incorrect datetime value: 'foo' for function str_to_date |
+---------+------+----------------------------------------------------------+
1 row in set (0.00 sec) ^ ^ just 1 single quote
Normally, the error message has a single set of single quotes. Yours has a double set, suggesting that you actually have a set of single quotes stored in your column data.
If this is the case, you can work around this by removing them where they exist:
SET datetimefile = (SELECT STR_TO_DATE(REPLACE(datestring,"'",''), '%c-%e-%Y %T'))
Using REPLACE() like this still would work even if not all of the rows contain the spurious quotes, since replace passes through the input value unchanged if the 'from_str' (2nd arg) doesn't occur.
From PHP SQL Request Correct:
LOAD DATA INFILE '.$filename."' INTO TABLE tablename (#var_DTime, `Product`, `Source`, `Cost`) SET `DTime` = str_to_date(#var_DTime,'%Y-%m-%dT%H:%i:%s')
Do not use: " " - 2010-12-31 01:48:28; - Don't work
Use "T" - 2010-12-31T01:48:28; - Work
I had a similar problem,
I wanted to use order by date syntax, but since my dates were in text format it returned the table unsorted.
I tried using
Alter table
Alter column `column name` date
but it gave the me same error you've got.
the only solution that I found by trial and error was to change the format of the date in the CSV file. I changed it to YYYY-MM-DD, then used "Alter table , alter column column name" to change the data type in the column
This error happened when you are trying to add a non-DateTime value inside a DateTime column.
For example, the DateTime format is YYYY-MM-DD HH:MM:SS. So in any case, if you try to add a date with any format other than the original format, it will throw this error.
In my case, I was using LOAD DATA INFILE to load a CSV file, to a specific table. One of the columns in the CSV file was in another date format, so I was supposed to change the format using str_to_date function using the below syntax:
LOAD DATA INFILE 'x.csv'
INTO TABLE orders FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(
timimg,
order_id,
order_details
)
set timing= STR_TO_DATE(lower(#timing), '%d-%b-%Y %H:%i:%s');
The above syntax throws the same error (Incorrect DateTime value for function str_to_date), So what was the issue?.
It was just adding the # for the timing column. # character means that you are telling MySQL to not load the data directly in the table for that column, however, make some changes to the value from CSV before inserting in the table.
So by adding the character # in the above query, It fixes the issue. The correct syntax is:
LOAD DATA INFILE 'x.csv'
INTO TABLE orders FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(
#timimg,
order_id,
order_details
)
set timing= STR_TO_DATE(lower(#timing), '%d-%b-%Y %H:%i:%s');
... SET datetimefile = STR_TO_DATE(datestring, '%c-%e-%Y-%T')
Note the lack of select around the str_to_date call. That select had no table reference, so the query failed with "unknown field datestring". That failure bubbled upwards and killed the entire overall query.