I've got a MySQL load script that almost works, it is perfect except for the date columns, which are not in a MySql friendly format.
load data infile '/Users/pfarrell/sandbox/waybase/folklore/Titles_1976.csv'
into table fix76
fields terminated by ','
enclosed by '"'
ignore 1 lines
( patentId, USPatentNum, title, grantDate, filedDate)
The problem is that my dates are in mm/dd/yyyy format. Looks like the str_to_date
function is what I want, but I can't figure out how to use it in the load command.
I'm envisioning something like:
grantDate = STR_TO_DATE(something, '%m/%d/%Y'),
but that doesn't work.
You can load the date strings into user-defined variables, and then use STR_TO_DATE(#date, '%m/%d/%Y') to convert them to MySQL dates.
Try this:
load data infile '/Users/pfarrell/sandbox/waybase/folklore/Titles_1976.csv'
into table fix76
fields terminated by ','
enclosed by '"'
ignore 1 lines
( patentId, USPatentNum, title, #grantDate, #filedDate)
set grantDate = STR_TO_DATE(#grantDate, '%m/%d/%Y'),
filedDate = STR_TO_DATE(#filedDate, '%m/%d/%Y')
Related
Hello I'm using LOAD DATA INFILE to populate a table in MySQL.
LOAD DATA INFILE 'test.txt'
INTO TABLE myTestTable
FIELDS TERMINATED BY '\t'
IGNORE 1 LINES;
Everything is working peachy except that there is a datetime column in my data that is formatted without any delimiter between the date and time sections. Like so:
SomeDateColumn
20050101081946
When I read this in, MySQL replaces all the dates with dummy values. Is there a way to have MySQL read this in correctly straight from a file?
Thanks!
You may call STR_TO_DATE when you run LOAD DATA, and convert the text date to a bona fide date on the fly:
LOAD DATA INFILE 'test.txt'
INTO TABLE myTestTable
FIELDS TERMINATED BY '\t'
IGNORE 1 LINES
(
col1, col2, #var1 -- list out all columns here
)
SET SomeDateColumn = STR_TO_DATE(#var1, '%Y%m%d%h%i%s');
I need a way to use the UTC_TIMESTAMP() function in a CSV file.
Currently I'm using the following Syntax;
LOAD DATA INFILE '/path/to/file.csv'
INTO TABLE my_table FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
I'm not quite sure what to do to keep the UTC_TIMESTAMP() from being enclosed in quotes. When entered in the database, I get the following result
"0000-00-00 00:00:00"
This was the only example I could find on stack overflow or on Google for converting a string to a MYSQL value.
MySQL CSV import: datetime value
I solved the problem by following the MySQL documentation on this page.
https://dev.mysql.com/doc/refman/5.7/en/load-data.html
About halfway down the page there is a section that shows you how to create variables for rows and then set table rows equal to native mysql functions or values assigned to those variables(Whichever you choose).
The example in the documentation that I'm referring to looks like this.
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(column1, column2)
SET column3 = CURRENT_TIMESTAMP;
I fixed my problem by restructuring my code like this...
LOAD DATA LOCAL INFILE '/path/to/file.csv'
INTO TABLE veh_icodes FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(id, vcode, year, make, model, body_style, tran_type, engine_cap, drive_train, doors, trim, created_by, updated_by, #created_at, #updated_at, active)
SET created_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP;"
I hope this helps someone. :)
I am trying to import a csv file to mysql table, But I need to remove First two characters on particular column before importing to mysql.
This is my statment :
string strLoadData = "LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets FIELDS terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES (SiteId,DateTime,Serial,DeviceId,AgentAID,VehicleRegistration,CarPark,SpaceNumber,GpsAddress,VehicleType,VehicleMake,VehicleModel,VehicleColour,IssueReasonCode,IssueReason,NoticeLocation,Points,Notes)";
Column IssueReasoncode' has data like 'LU12' , But i need to remove the first 2 characters it should have only integers on it and not alpha numeric .
I need to remove 'LU' from that column.
Is it possible to write like this on left(IssueReasonCode +' '2). This column is varchar(45) and cant be changed now because of large data on it.
Thanks
LOAD DATA INFILE has the ability to perform a function on the data for each column as you read it in (q.v. here). In your case, if you wanted to remove the first two characters from the IssueReasonCode column, you could use:
RIGHT(IssueReasonCode, CHAR_LENGTH(IssueReasonCode) - 2)
to remove the first two characters. You specify such column mappings at the end of the LOAD DATA statement using SET. Your statement should look something like the following:
LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets
FIELDS terminated by ','
ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(SiteId, DateTime, Serial, DeviceId, AgentAID, VehicleRegistration, CarPark, SpaceNumber,
GpsAddress, VehicleType, VehicleMake, VehicleModel, VehicleColour, IssueReasonCode,
IssueReason, NoticeLocation, Points, Notes)
SET IssueReasonCode = RIGHT(IssueReasonCode, CHAR_LENGTH(IssueReasonCode) - 2)
Referencing this and quoting this example , you can try the below to see if it works
User variables in the SET clause can be used in several ways. The
following example uses the first input column directly for the value
of t1.column1, and assigns the second input column to a user variable
that is subjected to a division operation before being used for the
value of t1.column2:
LOAD DATA INFILE 'file.txt' INTO TABLE t1 (column1, #var1) SET
column2 = #var1/100;
string strLoadData = "LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets FIELDS terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES (SiteId,DateTime,Serial,DeviceId,AgentAID,VehicleRegistration,CarPark,SpaceNumber,GpsAddress,VehicleType,VehicleMake,VehicleModel,VehicleColour,#IRC,IssueReason,NoticeLocation,Points,Notes) SET IssueReasonCode = substr(#IRC,2) ;";
I am using Load data in file query to insert csv into table. I have to format a date column inside the csv,
LOAD DATA INFILE '/invoices/invoice1381301986.csv' INTO TABLE invoice_table FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' (`code`,#var1) set datefield=STR_TO_DATE(#var1, '%m/%d/%Y');
I can format the date using the above query.
But the problem is, I have different formats for the csv date column. Possible formats are, "m/d/Y","m-d-Y", "m/d/y", "m-d-y", "Y-m-d", "Y/m/d".
so my query should be according to date format from the csv, so that I can modify my queries like,
datefield=STR_TO_DATE(#var1, '%m/%d/%Y')
OR
datefield=STR_TO_DATE(#var1, '%m-%d-%Y')..
How can I read the in which format the csv date field is?
you need to use a CASE :
LOAD DATA INFILE '/invoices/invoice1381301986.csv'
INTO TABLE invoice_table
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(`code`,#var1)
SET datefield= (
CASE
WHEN #var1 REGEXP '[0-9]{2}/[0-9]{2}/[0-9]{4}' THEN STR_TO_DATE(#var1,'%m/%d/%Y')
...
END
)
I have a table with payDate DATETIME in which I'm inserting using load data local infile 'file.txt' into table tableName; dates like 26/04/2012 00:00:00.
This gives warnings like Warning | 1265 | Data truncated for column 'payDate' at row 1 and the date in the table is 0000-00-00 00:00:00.
Is there any way to specify the format of the date?
Try STR_TO_DATE:
load data local infile 'file.txt' into table tableName
SET payDate = str_to_date(#payDate, '%d/%m/%Y');
I worked out without converting from datetime to varchar ,
Below is the working code for it -
mysql query to export csv data to local directory
SELECT organization_id,bank_name,branch_name,account_number,statement_type,parameter_name,
parameter_value,created_by,creation_date,updated_by,updation_date FROM cm_sub_param_values
INTO OUTFILE 'D:\mytable.csv'
FIELDS ESCAPED BY '""'
TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n';
In the csv exported , remove the column names and SAVE.
In the Table , creation_date and updation_date are of type datetime .
Import CSV data into mysql
LOAD DATA INFILE 'D:/mytable.csv' IGNORE
INTO TABLE `cm_sub_param_values`
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\''
LINES TERMINATED BY '\r\n'
(sub_param_id,organization_id,bank_name,branch_name,account_number,
statement_type,parameter_name,parameter_value,created_by,#creation_date,updated_by,#updation_date)
SET creation_date = STR_TO_DATE(#creation_date, '%m/%d/%Y %H:%i'),
updation_date = STR_TO_DATE(#updation_date, '%m/%d/%Y %H:%i') ;
Import Successful !
You would want to modify those variables before loading :
Link
set column type to VARCHAR
LOAD DATA INFILE
UPDATE SET column = DATE_FORMAT( str_to_date(column, '%d/%m/%Y'), '%Y-%m-%d' )
set column type back to DATE
SHOW WARNINGS