I have a csv file I'm trying to parse a bunch of strings and a few are date formats in my database. The code looks a little like this:
load data infile '/temp/db.csv'
into table db.dbtable
fields terminated by '|'
lines terminated by '\n'
ignore 1 rows
(db_id,
dbtable_field1,
db_date,
db_date2)
set
db_date = convert(char(10), Datetime),
db_date2 = convert(char(10), Datetime);
Every time I execute this it comes up with this error:
Error Code: 1292. Incorrect datetime value: '<date time>' for column 'db_date' at row 1
I've been using stackoverflow and other means to try to get this to work, but nothing seems to be working. Am I missing something important here?
--EDIT--
The csv file looks a little something like this:
db_id|dbtable_field1|db_date|db_date2
123|0025AAA|8/12/2014 10:24:32 AM| 8/12/2014 10:24:32 AM
124|0096AB54|2/26/2013|2/26/2013
125|0085ABDE|10/2/2014 4:56:00 PM|10/2/2014 4:56:00 PM
This block isn't exactly my csv file but it's the same format. As you can see the date format changes and thus I cannot use a predefined date format. Currently I am just parsing it as a varchar(25) in order to get it into the database simply, but parsing it into the date for further work would be useful later on (ie. date comparison testing).
I have done this in mutiple ways including:
db_date = str_to_date(#db_date, '%c/%d/%Y %h:%i:%s %p');
and
db_date = convert(char(10), getDate(), 125);
And neither have worked due to the inconsistency in date format.
Related
I am using the below sql to export data to a csv file:
select * INTO outfile 'customer.csv'
FIELDs terminated by ',' enclosed by '"'
LINES terminated by '\n'
from CUSTOMER customer
where customer.date_created < '2015-10-22 10:00:00';
I get this result in csv:
Problem is data doesn't import from this generated csv because the date format is different than in DB.
DB date format is yyyy-mm-dd hh:mm:ss. Also null values are replaced with \n which also fail when importing.
How can I generate the csv columns with correct i.e. yyyy-mm-dd hh:mm:ss date format and null/empty values?
Errors:
Incorrect datetime value: '23/07/2015 11:55' for column 'DATE_CREATED' at row 1
Incorrect integer value: '\N' for column 'column_name' at row 1
Note:
I am using mysql workbench to import the file.
I don't want to change the format/data directly in csv file.
Thanks
UPDATE:
Thanks to AdrianBR I realised I was opening the file with excel first which was overriding the date format hence wrong date format was showing even with notepad++.
\n is still a problem.
When opened with notepad++ for the first time it looks like this:
"100","0","2015-12-02 10:16:36","2015-12-02 10:16:36","0",\N,
The issue is most likely not coming from mysql.
It is most likely coming from the way Excel displays and later saves the dates.
to troublehsoot:
Open the file in a text editor such as notepad or notepad++ and check what the date looks like, if it's in ISO or not. It will probably be fine.
Now, if you open it in excel, it will be displayed in local format.
If you save the file now, you are likely to overwrite the ISO date format, with excel's local date format, making it not a valid importable mysql date anymore.
Moral: don't use excel when working with data, only use it to display charts. Excel makes assumptions about your data and messes with it in the most unexpected ways. Remember than 1.19 VAT tax rate? Excel seems to think it's the same as Jan 19. That integer ID? Excel thinks it's better off to write it in scientific notation and round it to first 4 digits. That Iso date? Excel thinks you are better off guessing which is the month and which is the date. That decimal point? surely you wanted comma as decimal, and dot as thousands separator instead. FTFY!
maybe specify the columns explicitly, and include the format in the select list like so:
TO_CHAR( mydate, 'yyyy-mm-dd hh24:mi:ss' )
edit:
as in:
SELECT my_id, my_val1, TO_CHAR( mydate, 'yyyy-mm-dd hh24:mi:ss' )
FROM mytable
I'm using a MySQL database with the Sequel Pro interface, and am new to SQL. I'm attempting to import data from a csv file and one of the columns I am importing into is of type datetime. However, the format I receive the data in is mm/dd/yy hh:mm AM/PM or null. Originally, I modified the type of the column to be varchar to avoid the issue but now I need to perform some date functions on the data that can't be done unless the column has a datetime type and format, so I need a way to convert the incoming data to the proper datetime format.
Additionally, people with no knowledge of SQL or databases are going to be running the import statement so it would be preferable to have them simply click file -- import and not have to enter anything complicated into the mysql command line. Also, after running a query I need to export the data in the same format it came in (mm/dd/yy hh:mm AM/PM or null).
Here are some sample values from the column:
Completion Time
null
6/16/14 10:33 AM
null
null
6/16/14 13:03 PM
6/17/14 13:53 PM
6/18/14 14:38 PM
6/18/14 14:52 PM
6/19/14 13:13 PM
6/18/14 18:56 PM
6/18/14 19:02 PM
null
A possibly simple solution that I've gathered might not be such a good idea from a couple of hours of googling, would be to keep the column type as varchar then somehow extract just the mm/dd/yy portion of the incoming data, convert that to proper MySQL date format and then perform my date functions.
Anyway any help would be greatly appreciated.
That's not very difficult with MySQLs date and time functions. STR_TO_DATE does what you need for the import:
the format I receive the data in is mm/dd/yy hh:mm AM/PM or null.
You get your DATETIME value with
STR_TO_DATE(yourValue, '%m/%d/%y %h:%i %p')
You find the specifiers for STR_TO_DATE at the description of the function DATE_FORMAT
For the export you do the reverse with the already mentioned function DATE_FORMAT with exact the same format string:
SELECT DATE_FORMAT(your_datetime_col, '%m/%d/%y %h:%i %p')
Have a look at this Demo
You can do the conversion at the INSERT statement like that:
INSERT INTO example (date_time) VALUES
(STR_TO_DATE('09/26/14 07:30 AM', '%m/%d/%y %h:%i %p'));
See it working in the updated Demo
Customizing an import with LOAD DATA INFILE
Let's have a table example with two columns id and date_time as
CREATE TABLE example (
id INT NOT NULL PRIMARY KEY,
date_time DATETIME
);
We have further a CSV file example.csv with data like that:
id,date
1,09/26/14 07:30 AM
2,07/23/14 07:30 PM
To import this file with LOAD DATA INFILE, you will use this statement:
LOAD DATA INFILE 'path/on/server/to/example.csv'
INTO TABLE example
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES -- because of the column headers in the first line
(id, #var1) -- You've got to map every field of your csv file to a column
-- of your table.
-- You've got to list the names of the columns of your table,
-- not the headers in the csv file.
-- if one field should be ignored, use another variable for this
-- field.
SET date_time = STR_TO_DATE(#var1, '%m/%d/%y %h:%i %p');
If your dates in the csv files contains the literal string 'null' that indicates a NULL value, then use the CASE operator:
date
09/26/14 07:30 AM
null
07/23/14 07:30 PM
then we've got to use
LOAD DATA INFILE 'path/on/server/to/example.csv'
INTO TABLE example
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n' -- your line endings
IGNORE 1 LINES -- because of the column headers in the first line
(#var1) -- read all parts of the date in variables
SET completionTime = CASE
WHEN #var1 = 'null' THEN NULL
ELSE STR_TO_DATE(#var1, '%m/%d/%y %h:%i %p')
END;
The problem with fields containing the separator, in this case the comma, you already solved with enclosing those fields (or simply all) with an enclosing character.
But we really should have a look at your real format.
The real answer to your question:
If you have untechnical people running Sequel Pro imports and need data manipulation, then you need to write and import/export script that users can upload and download an excel sheet from.
However...
I had a similar problem with importing dates with Sequel Pro. So here's a "half" solution. Take from it what you will.
(Note: this question was asked 5 years ago so this is for the benefit of anyone who runs across the question and runs into a similar Sequel Pro problem.)
This is a "half" solution:
Reformat the DateTime inside Excel first.
select column
Go to Format > Cells
Select "custom"
For type use "yyyy-mm-dd hh:mm:ss"
For 'null' values enter an obvious wrong date like "2099-01-01 00:00:00:00".
Import into Sequel Pro.
Open Sequel Pro
Go to View > Show Console (Do this to avoid a Sequel Pro CSV import crash)
Open your database connection and select the table.
Import the CSV.
Note any errors that occur
Clean up your data post-import.
(I told you this was a half solution)
In Sequel Pro run this query: UPDATE your_table SET your_column = NULL WHERE your_column = '2099-01-01 00:00:00:00';
i'm importing 1m+ records into my table from a csv file.
Works great using the load data local infile method.
However, the dates are all different formats.
A quick google lead me to this function:
STR_TO_DATE
However, when I implement that, I get nothing, an empty insert. here's my SQ cut down to include one date (I've 4 with the same issue) and generic column names:
load data local infile 'myfile.csv' into table `mytable`
fields terminated by '\t'
lines terminated by '\n'
IGNORE 1 LINES
( `column name 1`
, `my second column`
, #temp_date
, `final column`)
SET `Get Date` = STR_TO_DATE(#temp_date, '%c/%e/%Y')
If I do:
SET `Get Date` = #temp_date
The date from the csv is captured in the the format it was in the file.
However when I try the first method, my table column is empty. I've changed the column type to varchar (255) from timestamp to captre whatever is going in, but ultimatly, I want to capture y-m-d H:i:s (Not sure if STR_TO_DATE can do that?)
I'm also unsure as to why I need the # symbol.. google failed me there.
So, my questions are:
Why do I need the # symbol to use this function?
Should the data format ('%c/%e/%Y') be the format of the inputted data or my desired output?
Can I capture time in this way too?
sorry for the large post!
Back to Google for now...
Why do I need the # symbol to use this function?
The # symbol means that you are using a variable, so the read string isnt put right away into the table but into a memory pice that lets you operate with it before inserting it. More info in http://dev.mysql.com/doc/refman/5.0/en/user-variables.html
Should the data format ('%c/%e/%Y') be the format of the inputted data or my desired output?
Its the format of the inputted data, more info in http://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_str-to-date
Can I capture time in this way too?
You should be able to as long as you chose the correct format, something like
STR_TO_DATE(#temp_date,'%c/%e/%Y %h:%i:%s');
I had this problem. What solved it for me was making sure I accounted for whitespace that weren't delimiters in my load file. So if ',' is the delimiter:
..., 4/29/2012, ...
might be interpreted as " 4/29/2012"
So should be
...,4/29/2012,...
I have a csv file with a couple thousand game dates in it, but they are all in the MM/DD/YYYY format
2/27/2011,3:05 PM,26,14
(26 and 14 are team id #s), and trying to put them into SQL like that just results in 0000-00-00 being put into the date field of my table. This is the command I tried using:
LOAD DATA LOCAL INFILE 'c:/scheduletest.csv' INTO TABLE game
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(`date`, `time`, `awayteam_id`, `hometeam_id`);
but again, it wouldn't do the dates right. Is there a way I can have it convert the date as it tries to insert it? I found another SO question similar to this, but I couldn't get it to work.
Have you tried the following:
LOAD DATA LOCAL INFILE 'c:/scheduletest.csv' INTO TABLE game
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(#DATE_STR, `time`, `awayteam_id`, `hometeam_id`)
SET `date` = STR_TO_DATE(#DATE_STR, '%c/%e/%Y');
For more information, the documentation has details about the use of user variables with LOAD DATA (about half-way down - search for "User variables in the SET clause" in the page)
You can use variables to load the data from the csv into and run functions on them before inserting, like:
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(#datevar, #timevar, awayteam_id, hometeam_id)
SET date = STR_TO_DATE(#datevar, '%m/%d/%Y'),
SET time = etc etc etc;
My suggestion would be to insert the file into a temporary holding table where the date column is a character datatype. Then write a query with theSTR_TO_DATE conversion to move the data from the holding table to your final destination.
Convert field that you are using for the date to varchar type so it will play friendly with any format
Import CSV
Convert the dates to a valid mysql date format using something like:
UPDATE table SET field = STR_TO_DATE(field, '%c/%e/%Y %H:%i');
Then revert field type to date
Use a function to convert the format as needed.
I'm not an expert on MySQL, but http://dev.mysql.com/doc/refman/5.0/en/date-and-time-functions.html#function_str-to-date looks promising.
If you can't do that in the load command directly, you may try creating a table that allows you to load all the values as VARCHAR and then to do an insert into your game table with a select statement with the appropriate conversion instead.
If you file is not too big, you can use the Excel function TEXT. If, for example, your date is in cell A2, then the formula in a temporary column next to it would be =TEXT(A2,"yyyy-mm-dd hh:mm:ss"). This will do it and then you can paste the values of the formula's result back into the column and then delete the temporary column.
At csv file, the date field is in such format:
2/9/2010 7:32
3/31/2011 21:20
I am using php + mysql for development.
I need to read it and store into mysql db.
final value to store in mysql should be format as below:
2010-02-09 07:32:00
What's the correct way of it?
Is mysql syntax alone can handle the conversion easily?
Use the STR_TO_DATE() function.
Example
STR_TO_DATE('3/31/2011 21:20', '%c/%e/%Y %H:%i');
I face the same issue and after little research this is how i resolved it-
LOAD DATA LOCAL INFILE 'D:/dataupload.csv' INTO TABLE table1
FIELDS TERMINATED BY ',' ENCLOSED BY '' LINES TERMINATED BY '\r\n' (#d1,col2,col3,col4)
SET col1 = date_format(str_to_date(#d1, **'%m/%d/%Y'**), **'%Y-%m-%d'**)
Details:
'%m/%d/%Y' - this is the format of date in my CSV file
'%Y-%m-%d' - this is the mysql format in which i want to convert my CSV field date while inserting data
col1 - is the actual column of my table (having date data type)
#d1 - is the dummy variable to use in set statement, you can take it any variable
I had the same problem (with DATE) and another solution, is to use the native mysql format YYYYMMDD ie 20120209.
I haven't tried with DATETIME but I guess YYYYMMDDhhmmss will work.