MySQL: Load Data Infile from CSV that consist of comma in VARCHAR field - mysql

Below is the code that I use to import CSV file to MySQL database. It works well to divide all the field and its record.
LOAD DATA INFILE 'file.csv'
INTO TABLE customer FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n'
(
ID, name, salary, address, status
);
However, when there is a VARCHAR or TEXT field which consist of comma (','), it works improperly. It is because I use FIELDS TERMINATED BY ',' that used to separate each field record.
So, for example, if a customer with salary 50,000 (double), it split the field normally. But, if the customer address is Java Road 15, Hong Kong (varchar/text), Java Road 15 will be saved in address field, while the Hong Kong will be saved to status field. This basically remove any record inside the status field. Any clue for this problem? Thanks in advance.

Are the fields enclosed by double quotes or something else? If so, you can add the "ENCLOSED BY" in your query.
FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\r\n';
"Enclosed by" specifies the character to identify the start and end of a field. In your case, field is enclosed by a double quote such as "Java Road 15, Hong Kong". It helps MYSQL to extract the field correctly even if there is a field delimiter in the field.
MYSQL manual: https://dev.mysql.com/doc/refman/5.7/en/load-data.html

LOAD DATA INFILE 'file.csv'
INTO TABLE customer FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
(
ID, name, salary, address, status
);
Try this one.

If anyone happens to stumble upon this answer, I wanted to share b/c this took me way too long to figure out - using mac, datagrip, mysql 5.7+, survey response questions:
DROP TABLE IF EXISTS surveyQuestion_ID;
#create table
CREATE TABLE surveyQuestion_ID
(
surveyQuestion_ID INT(11) NOT NULL,
surveyDescription TEXT,
surveyResponse VARCHAR(25) DEFAULT NULL,
PRIMARY KEY (surveyQuestion_ID)
);
#load query
LOAD DATA LOCAL INFILE 'file_location/file.csv' INTO TABLE surveyQuestion_ID
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(surveyQuestion_ID, surveyDescription, surveyResponse);
Hope this helps.

Related

ignore first two characters on a column while importing csv to mysql

I am trying to import a csv file to mysql table, But I need to remove First two characters on particular column before importing to mysql.
This is my statment :
string strLoadData = "LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets FIELDS terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES (SiteId,DateTime,Serial,DeviceId,AgentAID,VehicleRegistration,CarPark,SpaceNumber,GpsAddress,VehicleType,VehicleMake,VehicleModel,VehicleColour,IssueReasonCode,IssueReason,NoticeLocation,Points,Notes)";
Column IssueReasoncode' has data like 'LU12' , But i need to remove the first 2 characters it should have only integers on it and not alpha numeric .
I need to remove 'LU' from that column.
Is it possible to write like this on left(IssueReasonCode +' '2). This column is varchar(45) and cant be changed now because of large data on it.
Thanks
LOAD DATA INFILE has the ability to perform a function on the data for each column as you read it in (q.v. here). In your case, if you wanted to remove the first two characters from the IssueReasonCode column, you could use:
RIGHT(IssueReasonCode, CHAR_LENGTH(IssueReasonCode) - 2)
to remove the first two characters. You specify such column mappings at the end of the LOAD DATA statement using SET. Your statement should look something like the following:
LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets
FIELDS terminated by ','
ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(SiteId, DateTime, Serial, DeviceId, AgentAID, VehicleRegistration, CarPark, SpaceNumber,
GpsAddress, VehicleType, VehicleMake, VehicleModel, VehicleColour, IssueReasonCode,
IssueReason, NoticeLocation, Points, Notes)
SET IssueReasonCode = RIGHT(IssueReasonCode, CHAR_LENGTH(IssueReasonCode) - 2)
Referencing this and quoting this example , you can try the below to see if it works
User variables in the SET clause can be used in several ways. The
following example uses the first input column directly for the value
of t1.column1, and assigns the second input column to a user variable
that is subjected to a division operation before being used for the
value of t1.column2:
LOAD DATA INFILE 'file.txt' INTO TABLE t1 (column1, #var1) SET
column2 = #var1/100;
string strLoadData = "LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets FIELDS terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES (SiteId,DateTime,Serial,DeviceId,AgentAID,VehicleRegistration,CarPark,SpaceNumber,GpsAddress,VehicleType,VehicleMake,VehicleModel,VehicleColour,#IRC,IssueReason,NoticeLocation,Points,Notes) SET IssueReasonCode = substr(#IRC,2) ;";

Tell sqlldr control file to load missing values as NULL

I have a CSV file. How can I tell the sqlldr control file to load missing values as NULL. (ie the table schema allows NULL for certain column)
Example of CSV
1,Name1
2,Name2
3,
4,Name3
Could you help me to edit my control file here so that a line 3 , the missing value is inserted as NULL in my table
Table
Create table test
( id Number(2), name Varchar(10), primary key (id) );
Control file
LOAD DATA INFILE '{path}\CSVfile.txt'
INSERT INTO test
FIELDS TERMINATED BY ','
(id CHAR,
name CHAR
)
I believe all you should have to do is this:
name CHAR(10) NULLIF(name=BLANKS)
You would have to hint to SQL*Loader that there might be nulls in your data.
2 ways to give that hint to SQL*Loader.
Use TRAILING NULLCOLS option.
LOAD DATA INFILE '{path}\CSVfile.txt'
INSERT INTO test<br>
FIELDS TERMINATED BY ','
TRAILING NULLCOLS
(id CHAR,
name CHAR
)
Recreate your CSV files with enclosed fields and then use OPTIONALLY ENCLOSED BY '"' which lets SQL*Loader clearly see the nulls in your data (nothing between quotes) that looks like "abcd",""
LOAD DATA INFILE '{path}\CSVfile.txt'
INSERT INTO test<br>
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
(id CHAR,
name CHAR
)
I found that using TRAILING NULLCOLS will do the job BUT it has to be for "blanks" at the end of the record line.
LOAD DATA INFILE {path}\Your_File
INSERT INTO TABLE Your_Table
TRAILING NULLCOLS
FIELDS TERMINATED BY ","
(
... your fields
)

Importing csv into MySQL

I've got a csv file I made with a bunch of info, but I cant get it to import properly...
Ive got these values in info.csv: id firstname lastname address state gpa credits
following a video, I used this:
LOAD DATA LOCAL INFILE '/IT101/info.csv' INTO TABLE 'student' FIELDS TERMINATED BY ','
ENCLOSED BY '"' ESCAPED BY '\' LINES TERMINATED BY '\n';
I get back:
PAGER set to stdout
and the values aren't there. What am I doing wrong?
EDIT: By the way, I alaready have two rows in the table from using insert into, just doing it 30 more times seemed like a waste of time
You can try...
LOAD DATA INFILE '/IT101/info.csv'
INTO TABLE students
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;

LOAD DATA INFILE id

I ran the following command:
LOAD DATA INFILE '/Users/Tyler/Desktop/players_20120318.txt' INTO TABLE players FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n';
On this data:
PlayerId,IsActive,IsVisible,FirstName,LastName,HeightFeet,HeightInches,Weight,Birthday,Gender,HometownCity,HometownState,HometownZip,HometownCountry,HighSchoolId,HighSchoolIdTemp,HighSchoolGradYear,CollegeYear,Redshirted,Transferred,CollegeId,CollegeIdTemp,CollegeGradYear,OtherAccountId,PreviousCollegeId,CurrentTeamId,LateralRecommendationReason,LateralRecommendationLink,CreationDate,CreatedBy,LastModifiedDate,LastModifiedBy,TwitterLink,FacebookLink,PersonalWebsite,PlayerImage,FirstNameNickName,NeulionID,OtherTeamID,OtherSportTypeID,SourceDataTypeID,PlayerTypeID,LoadID,SameNameTeammate,SameNameSchoolMate,SD_SportID,SD_PlayerID,ZeroNCAAStats,ModifiedByPythonGame,Missing2011,Transfer2011,RecruitingClass
21,True,True,John,Frost,6,1,185,,M,Decatur,AL,35603,,{A0AD8B45-47E1-4039-85DF-756301035073},7453,2009,JR,False,False,{299F909C-88D9-4D26-8ADC-3EC1A66168BB},844,2013,{EBA5A9E6-E03E-4AE5-B9B8-264339EE9259},,0,,,2011-02-16 20:53:34.877000000,,2012-03-08 01:43:37.593000000,{5EBB0160-E69A-4EA2-89D5-932DD4D58632},,,,,,,45759,1,1,5,,,,,,,,,,
1344,True,True,Zach,Alvord,6,0,173,,M,Alpharetta,GA,30022,,{379BF463-67A9-480E-8FFB-9B50AD494953},11597,2010,SO,False,False,{7208C8FB-6780-4379-BC25-5DC5064C85FD},36,2014,{CDACD2C7-7667-406C-9662-02B378B00032},,0,,,2011-02-16 20:53:34.970000000,,2012-03-07 23:28:17.343000000,{5EBB0160-E69A-4EA2-89D5-932DD4D58632},,,,,,,45710,1,1,5,,,,,,,,,,
And mySQL was taking that first column (PlayerID) and assigning it to the id column. It was also shifting everything over one column (first name was filled in with last name).
Is this the expected behavior?
I believe that MySQL will properly insert the data by skipping the id column as long as it's set to auto_increment. Otherwise you can specify the columns individually as Bobby pointed out.
To avoid this problem, specify the columns you're loading data into and leave out the id field:
LOAD DATA INFILE '/Users/Tyler/Desktop/players_20120318.txt' INTO TABLE players (col1, col2, col3...) FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n';

null values after csv import

Consider a table to store SSN and DOB.
CREATE TABLE fbi
(
ssn BIGINT,
dob DATE
)
Data was loaded into the table:
LOAD DATA LOCAL INFILE C:\test.csv
INTO TABLE fbi
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n';
SELECT * FROM fbi;
It is showing null values for DOB. I don't understand the error.
"ssn","dob"
5,"1952-11-15"
6,"1973-12-23"
6,"1951-12-23"
1,"1962-03-21"
It most likely has to do with the date formatting in your csv file. MySQL likes dates in the format yyyy-mm-dd. e.g. 2010-10-09. You might be able to get more information by issuing the following command in the MySQL command console immediately after your import:
show warnings;
UPDATE:
I see that your date field is quoted. If you have a quoted date, you'll need to tell MySQL that by adding OPTIONALLY ENCLOSED BY '"' to your import command (see MySQL manual for LOAD DATA INFILE). Try this:
LOAD DATA LOCAL INFILE C:\test.csv
INTO TABLE fbi
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n';
BTW: I hope those SSN's you've posted are fake or mangled in some unrecoverable way.