I have been banging my head against a wall trying to import datetime values from a .csv file.
Here's the import statement.
LOAD DATA LOCAL INFILE 'myData.csv'
INTO TABLE equity_last_import
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(equity,last,#last_date)
SET last_date = STR_TO_DATE( #last_date, '%Y-%m-%d %H:%i:%s')
Here's a sample of the data:
4108,48.74,"2013-09-16 16:15:04"
4249,8.1,"2013-09-16 16:15:04"
4197,3.81,"2013-09-16 17:20:00"
4139,26.81,"2013-09-16 16:15:04"
4218,24.83,"2013-09-16 17:20:00"
4260,79.72,"2013-09-16 16:15:04"
4270,450.12,"2013-09-16 17:20:00"
4242,30.38,"2013-09-16 16:15:04"
4193,1.42,"2013-09-16 16:15:04"
4134,3.77,"2013-09-16 16:15:04"
I am able to import date values using STR_TO_DATE() but I not able to get datetime values to import. I have tried several different date formats other than '%Y-%m-%d %H:%i:%s' and I always get a null datetime [0000-00-00 00:00:00]. I have also tried not using STR_TO_DATE(), since the string is in the default MySQL datetime format.
Any help will be appreciated.
I ran into the same problem. I fixed it by changing the format for the date column in my CSV file to match the MySQL datetime format.
Open CSV in Excel.
Highlight the column.
Right-click on the column.
Click on Format Cells.
Pick Custom.
Use yyyy/mm/dd hh:mm:ss in the Type field.
Click ok
My CSV successfully imported after I changed the datetime format as above.
The date in your data file is already in a format MySQL should natively understand. It's just enclosed in double quotes. You need to tell LOAD DATA INFILE how to deal with the quotes. Try something like this:
LOAD DATA LOCAL INFILE 'myData.csv'
INTO TABLE equity_last_import
FIELDS OPTIONALLY ENCLOSED BY '"' TERMINATED BY ','
LINES TERMINATED BY '\n'
(equity,last,last_date)
Update:
Since you've said it doesn't work, I created a test table and verified that it does work. Here's the proof:
I've highlighted your csv data from the question and pasted into a new file called myData.csv in my system's /tmp folder. Then I connected to the mysql console, switched to the test database and ran the following:
mysql> create table equity_last_import (equity int, last decimal(10,2), last_date datetime) engine=innodb;
Query OK, 0 rows affected (0.02 sec)
mysql> LOAD DATA LOCAL INFILE '/tmp/myData.csv'
-> INTO TABLE equity_last_import
-> FIELDS OPTIONALLY ENCLOSED BY '"' TERMINATED BY ','
-> LINES TERMINATED BY '\n'
-> (equity,last,last_date);
Query OK, 10 rows affected (0.00 sec)
Records: 10 Deleted: 0 Skipped: 0 Warnings: 0
mysql> select * from equity_last_import;
+--------+--------+---------------------+
| equity | last | last_date |
+--------+--------+---------------------+
| 4108 | 48.74 | 2013-09-16 16:15:04 |
| 4249 | 8.10 | 2013-09-16 16:15:04 |
| 4197 | 3.81 | 2013-09-16 17:20:00 |
| 4139 | 26.81 | 2013-09-16 16:15:04 |
| 4218 | 24.83 | 2013-09-16 17:20:00 |
| 4260 | 79.72 | 2013-09-16 16:15:04 |
| 4270 | 450.12 | 2013-09-16 17:20:00 |
| 4242 | 30.38 | 2013-09-16 16:15:04 |
| 4193 | 1.42 | 2013-09-16 16:15:04 |
| 4134 | 3.77 | 2013-09-16 16:15:04 |
+--------+--------+---------------------+
10 rows in set (0.00 sec)
See? It works perfectly.
Another Update:
You've specified that you're getting the following error now:
Out of range value for column 'last_date' at row 1
Does your CSV file have a header? If so, you may want to add IGNORE 1 LINES to your LOAD DATA INFILE command to tell MySQL to skip over the header.
Pulled my hair out over this also because I'm not importing in the above suggested way.
Workaround: Created a temporary field temp_date of type "VARCHAR" on your import table and have no problems loading the data. I then was able to perform an update on my date column which is of type date.
update table
set date = temp_date
I was having the same trouble and here's what I discovered, I think it might help you
The problem regards a GMT conflict:
The database I was extracting the .csv file was on GMT 00:00, so there wasn't daylight saving time.
My local server (which was the one I was trying to insert the .csv file) was running on my computer (system's) GMT, by default. In my case it was GMT -3, which is affected by daylight saving time.
SQL has a special way to deal with daylight saving time, it ignores the exact date and time when the daylight saving time starts to happen. In my case, it was october 20 and all the records between 00 and 1 AM of that day simply weren't recognized by the server, even if the format was correct, because they simply 'didn't exist' (see more here). This was affecting all timestamp records, not only the specifics of daylight saving time.
My solution was to set the local server to GMT +00, before creating the new table and import the .csv, using
SET time_zone='+00:00';
And then when I imported the .csv file all the records were read properly. I think if you set the time zone equal to the one that generated the .csv file should work!
Open CSV in Excel.
Highlight the column.
Right-click on the column.
Click on Format Cells.
Pick Custom.
Use same format as SQL date format yyyy-mm-dd
Click ok and save
it's working fine for me
Related
Even though I know aws has mentioned on their documentation that csv is more like txt file for them. But why there is no entry for CSV file.
For example:
If I am running a query like:
COPY "systemtable" FROM 's3://test/example.txt' <credentials> IGNOREHEADER 1 delimiter as ','
then its creating entry in stl_load_commits, which I can query by:
select query, curtime as updated from stl_load_commits where query = pg_last_copy_id();
But, in same way when I am tryig with:
COPY "systemtable" FROM 's3://test/example.csv'
<credentials> IGNOREHEADER 1 delimiter as ',' format csv;
then return from
select query, curtime as updated from stl_load_commits where query = pg_last_copy_id();
is blank, Why aws does not create entry for csv ?
This is the first part of the question. Secondly, there must be some way through which we can check the status of the loaded file?
How can we check if the file has successfully loaded in DB if the file is of type csv?
The format of the file does not affect the visibility of success or error information in system tables.
When you run COPY it returns confirmation of success and a count of rows loaded. Some SQL clients may not return this information to you but here's what it looks like using psql:
COPY public.web_sales from 's3://my-files/csv/web_sales/'
FORMAT CSV
GZIP
CREDENTIALS 'aws_iam_role=arn:aws:iam::01234567890:role/redshift-cluster'
;
-- INFO: Load into table 'web_sales' completed, 72001237 record(s) loaded successfully.
-- COPY
If the load succeeded you can see the files in stl_load_commits:
SELECT query, TRIM(file_format) format, TRIM(filename) file_name, lines, errors FROM stl_load_commits WHERE query = pg_last_copy_id();
query | format | file_name | lines | errors
---------+--------+---------------------------------------------+---------+--------
1928751 | Text | s3://my-files/csv/web_sales/0000_part_03.gz | 3053206 | -1
1928751 | Text | s3://my-files/csv/web_sales/0000_part_01.gz | 3053285 | -1
If the load fails you should get an error. Here's an example error (note the table I try to load):
COPY public.store_sales from 's3://my-files/csv/web_sales/'
FORMAT CSV
GZIP
CREDENTIALS 'aws_iam_role=arn:aws:iam::01234567890:role/redshift-cluster'
;
--ERROR: Load into table 'store_sales' failed. Check 'stl_load_errors' system table for details.
You can see the error details in stl_load_errors.
SELECT query, TRIM(filename) file_name, TRIM(colname) "column", line_number line, TRIM(err_reason) err_reason FROM stl_load_errors where query = pg_last_copy_id();
query | file_name | column | line | err_reason
---------+------------------------+-------------------+------+---------------------------
1928961 | s3://…/0000_part_01.gz | ss_wholesale_cost | 1 | Overflow for NUMERIC(7,2)
1928961 | s3://…/0000_part_02.gz | ss_wholesale_cost | 1 | Overflow for NUMERIC(7,2)
This is sql file
# mo.sql
DROP TABLE IF EXISTS mo;
## _CREATE_TABLE_
CREATE TABLE mo
(
name CHAR(30),
age INT,
salary INT
);
## _CREATE_TABLE_
LOAD DATA LOCAL INFILE 'moja-2001.txt' INTO TABLE mo;
I run it from Linux terminal
mysql -p cookbook < mo.sql
No warning,no errors.
SELECT * FROM mo;
+--------------------------+------+--------+
| name | age | salary |
+--------------------------+------+--------+
| jova jovic | 24 | NULL |
| ceda prashak | 25 | NULL |
| toma grobar 28 20001 | NULL | NULL |
+--------------------------+------+--------+
I have created txt file with geany text editor
jova jovic 24 999
ceda prashak 25 1000
toma grobar 28 20001
Why is salary column wrong?Why is third row wrong also?
The last row does not use the correct separators for values. I suspect the other rows separate the values using tabs and the last one uses spaces. The same for column salary.
Make sure you use the same separator for values. It's better to use comma (it is visible and less error prone) and use the FIELDS TERMINATED BY clause of the LOAD DATA statement to inform MySQL about it.
Change the file to look like this:
jova jovic,24,999
ceda prashak,25,1000
toma grobar,28,20001
and import it like this:
LOAD DATA LOCAL INFILE 'moja-2001.txt' INTO TABLE mo FIELDS TERMINATED BY ',';
Read more about the LOAD DATA statement.
can someone help me with this, my resulting table is showing only zeros for the timestamp. I tried changing the field type to datatime and timestamp but not luck
MYSQL
LOAD DATA LOCAL INFILE 'vals.csv'
INTO TABLE vals
FIELDS TERMINATED BY ","
LINES TERMINATED BY "\n"
IGNORE 1 LINES
(#varTimeSt,NOMINAL,NAME,ID,VAL) SET DATETIME = STR_TO_DATE(#varTimeSt,'%d/%m/%Y %h:%i:%s');
CSV File
DATETIME,NAME,ID,VAL
"25/08/2016 02:00:00",tom,cNum6,12
"25/08/2016 02:00:00",Charles,cNum7,10.58
"25/08/2016 02:00:00",Donal,cNum8,10.18
"25/08/2016 02:00:00",Duncan,cNum7,10.31
Resulting table
DATETIME,NAME,ID,VAL
0000-00-00 00:00:00,tom,cNum6,12
0000-00-00 00:00:00,Charles,cNum7,10.58
0000-00-00 00:00:00,Donal,cNum8,10.18
0000-00-00 00:00:00,Duncan,cNum7,10.31
Your problem is that your dates are not in native mysql DATETIME format (%Y-%m-%d %H:%i:%s), this can be shown as such:
mysql> SELECT CAST('25/08/2016 02:00:00' AS DATETIME);
+-----------------------------------------+
| CAST('25/08/2016 02:00:00' AS DATETIME) |
+-----------------------------------------+
| NULL |
+-----------------------------------------+
1 row in set, 1 warning (0,00 sec)
mysql> SHOW WARNINGS;
+---------+------+-------------------------------------------------+
| Level | Code | Message |
+---------+------+-------------------------------------------------+
| Warning | 1292 | Incorrect datetime value: '25/08/2016 02:00:00' |
+---------+------+-------------------------------------------------+
1 row in set (0,00 sec)
The solution is to get them into MySQL format, this could be done by manipulating your DATETIME column on disk or at source. This may be more difficult than the 'self sufficient' solution I'll outline below.
There exists the string to date (STR_TO_DATE) function to solve this kind of issue once data is inside the database, here is a demonstration with your format. Otherwise see the manual here. (See the DATE_FORMAT section for a table with the codes you can use for different date formats.):
mysql> SELECT CAST(STR_TO_DATE('25/08/2016 02:00:00', '%d/%m/%Y %H:%i:%s') AS DATETIME);
+---------------------------------------------------------------------------+
| CAST(STR_TO_DATE('25/08/2016 02:00:00', '%d/%m/%Y %H:%i:%s') AS DATETIME) |
+---------------------------------------------------------------------------+
| 2016-08-25 02:00:00 |
+---------------------------------------------------------------------------+
1 row in set (0,00 sec)
Personally in this situation I'd load the data into a temporary table that accepted the date fields as a string and then use an INSERT with the STR_TO_DATEfunction to get the data into the final table. A final solution (untested, guideline only) would look like this:
CREATE TEMPORARY TABLE valsLoad (DATETIME TEXT,NAME TEXT,ID TEXT,VAL INT);
LOAD DATA LOCAL INFILE 'vals.csv'
INTO TABLE valsLoad
FIELDS TERMINATED BY ","
LINES TERMINATED BY "\n"
IGNORE 1 LINES
(#varTimeSt,NOMINAL,NAME,ID,VAL) SET DATETIME = STR_TO_DATE(#varTimeSt,'%d/%m/%Y %h:%i:%s');
INSERT INTO vals
SELECT STR_TO_DATE(DATETIME, '%d/%m/%Y %H:%i:%m'), NAME, ID, VAL FROM valsLoad;
DROP TEMPORARY TABLE valsLoad;
Please let me know if you have any questions.
Regards,
James
I have a .csv file that contains 5 columns of data. Occasionally, the last column contains a NULL value, like so.
2014-07-11 23:55:00,1,245,0.05,0.01,0.0003
2014-07-11 23:57:00,1,245,0.05,0.01,\N
2014-01-17 20:14:00,2,215,0.05,0.009,0.002
I'm attempting to load this into a local database, so from the MySQL console I run the following code:
LOAD DATA LOCAL INFILE 'C:/<redacted>/data.csv'
INTO TABLE tbl_data
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
This method does work just fine when the column that contains the NULL value is moved so that it isn't in the last column, but it throws warnings back at me when the data is formatted as shown above.
Query OK, 71735 rows affected, 49 warnings
' for column 'energy' at row 5253 | value: 'N
I followed advice from this thread, and attempted to utilize a local variable, but that didn't appease the MySQL gods either.
LOAD DATA LOCAL INFILE 'C:/<redacted>/data.csv'
INTO TABLE tbl_data
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
(time, device, voltage, amps, power, #energy)
SET energy = nullif(#energy,'\N')
Oh, and the table structure is set up like so, and I'm running MySQL 5.6.17:
+--------+-------------+-----+---------+
|NAME |TYPE |NULL | DEFAULT |
|time |datetime |No | None |
|device |tinyint(1) |No | None |
|voltage |smallint(3) |No | None |
|amps |decimal(6,3) |No | None |
|power |decimal(6,3) |No | None |
|energy |decimal(7,4) |Yes | NULL |
+--------+-------------+-----+---------+
What am I doing wrong here??
Rimas was correct, the last clause in my initial statement should have been:
LINES TERMINATED BY '\r\n'
The relevant documentation is worded as follows:
If you have generated the text file on a Windows system, you might have to use LINES TERMINATED BY '\r\n' to read the file properly, because Windows programs typically use two characters as a line terminator. Some programs, such as WordPad, might use \r as a line terminator when writing files. To read such files, use LINES TERMINATED BY '\r'.
What's the right syntax to insert a value inside a column of type bit(1) in `MySQL'?
My column definition is:
payed bit(1) NOT NULL
I'm loading the data from a csv where the data is saved as 0 or 1.
I've tried to do the insert using:
b'value' or 0bvalue (example b'1' or 0b1)
As indicated from the manual.
But I keep getting this error:
Warning | 1264 | Out of range value for column 'payed' at row 1
What's the right way to insert a bit value?
I'm not doing the insert manually but I'm loading the data from a csv (using load data infile) in which the data for the column is 0 or 1.
This is my load query, I've renamed the fields for privacy questions, there's no error in that definition:
load data local infile 'input_data.csv' into table table
fields terminated by ',' lines terminated by '\n'
(id, year, field1, #date2, #date1, field2, field3, field4, field5, field6, payed, field8, field9, field10, field11, project_id)
set
date1 = str_to_date(#date1, '%a %b %d %x:%x:%x UTC %Y'),
date2 = str_to_date(#date2, '%a %b %d %x:%x:%x UTC %Y');
show warnings;
This is an example row of my CSV:
200014,2013,0.0,Wed Feb 09 00:00:00 UTC 2014,Thu Feb 28 00:00:00 UTC 2013,2500.0,21,Business,0,,0,40.0,0,PROSPECT,1,200013
Update:
I didn't find a solution with the bit, so I've changed the column data type from bit to tinyint to make it work.
I've finally found the solution and I'm posting it here for future reference. I've found help in the mysql load data manual page.
So for test purpose my table structure is:
+--------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------+-------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| nome | varchar(45) | YES | | NULL | |
| valore | bit(1) | YES | | NULL | |
+--------+-------------+------+-----+---------+-------+
My csv test file is:
1,primo_valore,1
2,secondo_valore,0
3,terzo_valore,1
The query to load the csv into the table is:
load data infile 'test.csv' into table test
fields terminated by ',' lines terminated by '\n'
(id, nome, #valore) set
valore=cast(#valore as signed);
show warnings;
As you can see do load the csv you need to do a cast cast(#valore as signed) and in your csv you can use the integer notation 1 or 0 to indicate the bit value. This is because BIT values cannot be loaded using binary notation (for example, b'011010').
Replace the "0" values in the csv by no value at all. That worked for me.
You can use BIN() function like this :
INSERT INTO `table` VALUES (`column` = BIN(1)), (`column` = BIN(0));
Let me guess, but I think you should ignore 1st line of your CSV file in LOAD query.
See "IGNORE number LINES"