I am using Load data in file query to insert csv into table. I have to format a date column inside the csv,
LOAD DATA INFILE '/invoices/invoice1381301986.csv' INTO TABLE invoice_table FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' (`code`,#var1) set datefield=STR_TO_DATE(#var1, '%m/%d/%Y');
I can format the date using the above query.
But the problem is, I have different formats for the csv date column. Possible formats are, "m/d/Y","m-d-Y", "m/d/y", "m-d-y", "Y-m-d", "Y/m/d".
so my query should be according to date format from the csv, so that I can modify my queries like,
datefield=STR_TO_DATE(#var1, '%m/%d/%Y')
OR
datefield=STR_TO_DATE(#var1, '%m-%d-%Y')..
How can I read the in which format the csv date field is?
you need to use a CASE :
LOAD DATA INFILE '/invoices/invoice1381301986.csv'
INTO TABLE invoice_table
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(`code`,#var1)
SET datefield= (
CASE
WHEN #var1 REGEXP '[0-9]{2}/[0-9]{2}/[0-9]{4}' THEN STR_TO_DATE(#var1,'%m/%d/%Y')
...
END
)
Related
I've got a MySQL load script that almost works, it is perfect except for the date columns, which are not in a MySql friendly format.
load data infile '/Users/pfarrell/sandbox/waybase/folklore/Titles_1976.csv'
into table fix76
fields terminated by ','
enclosed by '"'
ignore 1 lines
( patentId, USPatentNum, title, grantDate, filedDate)
The problem is that my dates are in mm/dd/yyyy format. Looks like the str_to_date
function is what I want, but I can't figure out how to use it in the load command.
I'm envisioning something like:
grantDate = STR_TO_DATE(something, '%m/%d/%Y'),
but that doesn't work.
You can load the date strings into user-defined variables, and then use STR_TO_DATE(#date, '%m/%d/%Y') to convert them to MySQL dates.
Try this:
load data infile '/Users/pfarrell/sandbox/waybase/folklore/Titles_1976.csv'
into table fix76
fields terminated by ','
enclosed by '"'
ignore 1 lines
( patentId, USPatentNum, title, #grantDate, #filedDate)
set grantDate = STR_TO_DATE(#grantDate, '%m/%d/%Y'),
filedDate = STR_TO_DATE(#filedDate, '%m/%d/%Y')
I am trying to import a csv file to mysql table, But I need to remove First two characters on particular column before importing to mysql.
This is my statment :
string strLoadData = "LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets FIELDS terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES (SiteId,DateTime,Serial,DeviceId,AgentAID,VehicleRegistration,CarPark,SpaceNumber,GpsAddress,VehicleType,VehicleMake,VehicleModel,VehicleColour,IssueReasonCode,IssueReason,NoticeLocation,Points,Notes)";
Column IssueReasoncode' has data like 'LU12' , But i need to remove the first 2 characters it should have only integers on it and not alpha numeric .
I need to remove 'LU' from that column.
Is it possible to write like this on left(IssueReasonCode +' '2). This column is varchar(45) and cant be changed now because of large data on it.
Thanks
LOAD DATA INFILE has the ability to perform a function on the data for each column as you read it in (q.v. here). In your case, if you wanted to remove the first two characters from the IssueReasonCode column, you could use:
RIGHT(IssueReasonCode, CHAR_LENGTH(IssueReasonCode) - 2)
to remove the first two characters. You specify such column mappings at the end of the LOAD DATA statement using SET. Your statement should look something like the following:
LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets
FIELDS terminated by ','
ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(SiteId, DateTime, Serial, DeviceId, AgentAID, VehicleRegistration, CarPark, SpaceNumber,
GpsAddress, VehicleType, VehicleMake, VehicleModel, VehicleColour, IssueReasonCode,
IssueReason, NoticeLocation, Points, Notes)
SET IssueReasonCode = RIGHT(IssueReasonCode, CHAR_LENGTH(IssueReasonCode) - 2)
Referencing this and quoting this example , you can try the below to see if it works
User variables in the SET clause can be used in several ways. The
following example uses the first input column directly for the value
of t1.column1, and assigns the second input column to a user variable
that is subjected to a division operation before being used for the
value of t1.column2:
LOAD DATA INFILE 'file.txt' INTO TABLE t1 (column1, #var1) SET
column2 = #var1/100;
string strLoadData = "LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets FIELDS terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES (SiteId,DateTime,Serial,DeviceId,AgentAID,VehicleRegistration,CarPark,SpaceNumber,GpsAddress,VehicleType,VehicleMake,VehicleModel,VehicleColour,#IRC,IssueReason,NoticeLocation,Points,Notes) SET IssueReasonCode = substr(#IRC,2) ;";
I've a CSV file where values may have comma as part of value
"09200, France, Paris", "Tower ""Olivia"""
"09200, Spain, Barselona", Shop - perfect
However once I import data it breaks value on 4 columns (based on number of comma in row). What do I do wrong? Please see my import command below.
LOAD DATA LOCAL INFILE '~/Downloads/file.csv' INTO TABLE my_table CHARACTER SET utf8 FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (#col1,#col2) set address=#col1,name=#col2;
Add an ENCLOSED BY clause to your query:
LOAD DATA LOCAL INFILE '~/Downloads/file.csv'
INTO TABLE my_table
CHARACTER SET utf8
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(#col1,#col2) set address=#col1,name=#col2;
i am loading data from CSV file,it contains around 70k records ,in csv file date format for one column is like 10/24/2013 5:27:05 PM and in my table date formate is like(datetime 00-00-0000) ,so when i am uploading data form csv file using below command it was failed due to different time stamp.
LOAD DATA INFILE "/tmp/fulldata.csv" INTO TABLE test.customerorders COLUMNS TERMINATED BY ',' LINES TERMINATED BY '\n';
how can fix this time stamp in my table and how can i upload data in to the table?
You could convert the data like following:
SELECT STR_TO_DATE( '10/24/2013 5:27:05 PM', '%m/%d/%Y %I:%i:%s %p' )
and store it in a datetime field
edit:
LOAD data file can load pretty faster for 70K records.
LOAD DATA LOCAL INFILE '/filelocationfromroot' INTO TABLE `yourtablename`
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(
`tablefieldforfirstcsvcolumn`,`tablefieldforsecondcsvcolumn`,...,#yourdatecolumn,
)
SET `datacolumnname`=STR_TO_DATE( #yourdatecolumn, '%m/%d/%Y %I:%i:%s %p' )
Consider a table to store SSN and DOB.
CREATE TABLE fbi
(
ssn BIGINT,
dob DATE
)
Data was loaded into the table:
LOAD DATA LOCAL INFILE C:\test.csv
INTO TABLE fbi
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n';
SELECT * FROM fbi;
It is showing null values for DOB. I don't understand the error.
"ssn","dob"
5,"1952-11-15"
6,"1973-12-23"
6,"1951-12-23"
1,"1962-03-21"
It most likely has to do with the date formatting in your csv file. MySQL likes dates in the format yyyy-mm-dd. e.g. 2010-10-09. You might be able to get more information by issuing the following command in the MySQL command console immediately after your import:
show warnings;
UPDATE:
I see that your date field is quoted. If you have a quoted date, you'll need to tell MySQL that by adding OPTIONALLY ENCLOSED BY '"' to your import command (see MySQL manual for LOAD DATA INFILE). Try this:
LOAD DATA LOCAL INFILE C:\test.csv
INTO TABLE fbi
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n';
BTW: I hope those SSN's you've posted are fake or mangled in some unrecoverable way.