Import CSV into MySQL - Offset by 1 Column - mysql

I am importing my data from a csv file like this:
LOAD DATA LOCAL INFILE "c:\myfile.csv" INTO TABLE historic FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n';
This ALMOST works perfect. My table has one different row, the first: An auto incremented id column. Otherwise the csv file and the MySQL table match perfectly. When I run the command above, it puts the first column from the csv into the id field. It want it to actually go into the second.
Is there a way I can modify the above statement to either specify the columns or just offset it by 1? (skipping the first column on import).


You can optionally name the columns to be populated by LOAD DATA, and simply omit your id column:
LOAD DATA LOCAL INFILE "c:\myfile.csv" INTO TABLE historic
FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n'
(column2, colum3, column4, column5, ...)


You can load your data specifing the order columns that you're going to use into your table:
LOAD DATA LOCAL FILE '/tmp/test.txt' INTO TABLE historic
(name_field1, name_field2, name_field3...);
Or if you prefer you can load it first to a temporary table and use a select into statement to laod it into your final table (it's slower).

Related

ignore first two characters on a column while importing csv to mysql

I am trying to import a csv file to mysql table, But I need to remove First two characters on particular column before importing to mysql.
This is my statment :
string strLoadData = "LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets FIELDS terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES (SiteId,DateTime,Serial,DeviceId,AgentAID,VehicleRegistration,CarPark,SpaceNumber,GpsAddress,VehicleType,VehicleMake,VehicleModel,VehicleColour,IssueReasonCode,IssueReason,NoticeLocation,Points,Notes)";
Column IssueReasoncode' has data like 'LU12' , But i need to remove the first 2 characters it should have only integers on it and not alpha numeric .
I need to remove 'LU' from that column.
Is it possible to write like this on left(IssueReasonCode +' '2). This column is varchar(45) and cant be changed now because of large data on it.
Thanks

LOAD DATA INFILE has the ability to perform a function on the data for each column as you read it in (q.v. here). In your case, if you wanted to remove the first two characters from the IssueReasonCode column, you could use:
RIGHT(IssueReasonCode, CHAR_LENGTH(IssueReasonCode) - 2)
to remove the first two characters. You specify such column mappings at the end of the LOAD DATA statement using SET. Your statement should look something like the following:
LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets
FIELDS terminated by ','
ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(SiteId, DateTime, Serial, DeviceId, AgentAID, VehicleRegistration, CarPark, SpaceNumber,
GpsAddress, VehicleType, VehicleMake, VehicleModel, VehicleColour, IssueReasonCode,
IssueReason, NoticeLocation, Points, Notes)
SET IssueReasonCode = RIGHT(IssueReasonCode, CHAR_LENGTH(IssueReasonCode) - 2)

Referencing this and quoting this example , you can try the below to see if it works
User variables in the SET clause can be used in several ways. The
following example uses the first input column directly for the value
of t1.column1, and assigns the second input column to a user variable
that is subjected to a division operation before being used for the
value of t1.column2:
LOAD DATA INFILE 'file.txt' INTO TABLE t1 (column1, #var1) SET
column2 = #var1/100;
string strLoadData = "LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets FIELDS terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES (SiteId,DateTime,Serial,DeviceId,AgentAID,VehicleRegistration,CarPark,SpaceNumber,GpsAddress,VehicleType,VehicleMake,VehicleModel,VehicleColour,#IRC,IssueReason,NoticeLocation,Points,Notes) SET IssueReasonCode = substr(#IRC,2) ;";

Import csv file with less columns than the table to which I'm importing?

I have exported a csv file with four columns in phpMyAdmin whith these options:
Columns separated with: ,
Columns enclosed with: "
Columns escaped with: "
Lines terminated with: AUTO
Now I want to import this file into a table with 17 columns!?
So only four columns will be populated and the others will stay empty.

You can spcify the column name while import data from file, and use user variable to discard an input value by assigning it to a user variable while import data in table .
AS PER MYSQL DOCUMENTATION
You can also discard an input value by assigning it to a user variable
and not assigning the variable to a table column:
LOAD DATA INFILE 'file.txt' INTO TABLE t1 (column1, #dummy,
column2, #dummy, column3);
LOAD DATA INFILE 'filename.csv'
INTO TABLE sqlTable(column1, column2, column3, column4, #NotInsert2, #NotInsert2... till #NotInsert13);
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
So in Your case, Just specify four column name, and use any user variable instead of others column name so that total number of column in list will be equal to 17.
EDIT - Lets understand with Example:-
Consider we have to load file with following contents:
File-name: example.csv
col-2,col-3,col-4
a,2,3
b,4,5
c,6,7
Case :- When csv file have lesser number of columns than targeted table
table structure: tablename
col-1 col-2 col-3 col-4
Consider, col-1 is auto-increment and not provided in csv.
LOAD DATA INFILE ‘path/to/example.csv’
INTO TABLE tablename
FIELDS TERMINATED BY ‘,’
LINES
TERMINATED BY ‘\n’
IGNORE 1 LINES (col-2,col-3,col-4) set col-1=null;
Passing null value will make col-1 to take an auto-increment value.
Using SET you can assign values to those columns which were not available in csv and are not null.
For more information refer here

Try this:
LOAD DATA LOCAL INFILE 'XYZ.csv'
INTO TABLE tickets
FIELDS terminated by ','
ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES
(column1, column2, column3, column4)(choose your data columns) ";

Import CSV to MySQL With Fewer Columns Than Destination Table

Is there a way to import a CSV into MySQL where the column numbers do not match?
I have a MySQL "Employees" table which we update once a month with a CSV file. I have added two additional columns to the "Employees" table that are relevant to our needs. Importing the table data, through MySQL Workbench, with the following SQL statement:
LOAD DATA LOCAL INFILE 'EmployeeDump-March.csv' REPLACE INTO TABLE employees
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
I receive the following error, which is absolutely correct... the column numbers don't match:
Error Code: 1261. Row 1 doesn't contain data for all columns 7.426 sec
The two extra fields have default values.
Is there a way to supress this error and have MySQL ignore those last two columns, leaving them alone?

You can (and should!) explicitly name the columns you are inserting into using LOAD DATA INFILE, just like a regular INSERT. You can also optionally set default values for the two new columns you added as part of the LOAD DATA INFILE statement.
Something like this. Just fill in the actual column names in the right order between the parens, and set the actual defaults and column names for the new columns if you don't want NULL:
LOAD DATA LOCAL INFILE 'EmployeeDump-March.csv' REPLACE INTO TABLE employees
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(old_column1, old_column2, ...)
set new_column1 = NULL,
new_column2 = NULL

Insert CSV files based on field name

Now am using the below code to upload CSV files into MYSQL database via FTP.It's Working fine.The CSV column have a field name "STATUS". It's have two values A and D. But i want to insert particular rows only based on field "STATUS" A. How can i insert this?.
LOAD DATA LOCAL INFILE '/root/782012_10.csv'
INTO TABLE tbl_dndno
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\n';

You could load the file into a temporary table, then insert from the temporary table into your main table with a query that selects the rows you want.

LOAD DATA INFILE id

I ran the following command:
LOAD DATA INFILE '/Users/Tyler/Desktop/players_20120318.txt' INTO TABLE players FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n';
On this data:
PlayerId,IsActive,IsVisible,FirstName,LastName,HeightFeet,HeightInches,Weight,Birthday,Gender,HometownCity,HometownState,HometownZip,HometownCountry,HighSchoolId,HighSchoolIdTemp,HighSchoolGradYear,CollegeYear,Redshirted,Transferred,CollegeId,CollegeIdTemp,CollegeGradYear,OtherAccountId,PreviousCollegeId,CurrentTeamId,LateralRecommendationReason,LateralRecommendationLink,CreationDate,CreatedBy,LastModifiedDate,LastModifiedBy,TwitterLink,FacebookLink,PersonalWebsite,PlayerImage,FirstNameNickName,NeulionID,OtherTeamID,OtherSportTypeID,SourceDataTypeID,PlayerTypeID,LoadID,SameNameTeammate,SameNameSchoolMate,SD_SportID,SD_PlayerID,ZeroNCAAStats,ModifiedByPythonGame,Missing2011,Transfer2011,RecruitingClass
21,True,True,John,Frost,6,1,185,,M,Decatur,AL,35603,,{A0AD8B45-47E1-4039-85DF-756301035073},7453,2009,JR,False,False,{299F909C-88D9-4D26-8ADC-3EC1A66168BB},844,2013,{EBA5A9E6-E03E-4AE5-B9B8-264339EE9259},,0,,,2011-02-16 20:53:34.877000000,,2012-03-08 01:43:37.593000000,{5EBB0160-E69A-4EA2-89D5-932DD4D58632},,,,,,,45759,1,1,5,,,,,,,,,,
1344,True,True,Zach,Alvord,6,0,173,,M,Alpharetta,GA,30022,,{379BF463-67A9-480E-8FFB-9B50AD494953},11597,2010,SO,False,False,{7208C8FB-6780-4379-BC25-5DC5064C85FD},36,2014,{CDACD2C7-7667-406C-9662-02B378B00032},,0,,,2011-02-16 20:53:34.970000000,,2012-03-07 23:28:17.343000000,{5EBB0160-E69A-4EA2-89D5-932DD4D58632},,,,,,,45710,1,1,5,,,,,,,,,,
And mySQL was taking that first column (PlayerID) and assigning it to the id column. It was also shifting everything over one column (first name was filled in with last name).
Is this the expected behavior?

I believe that MySQL will properly insert the data by skipping the id column as long as it's set to auto_increment. Otherwise you can specify the columns individually as Bobby pointed out.

To avoid this problem, specify the columns you're loading data into and leave out the id field:
LOAD DATA INFILE '/Users/Tyler/Desktop/players_20120318.txt' INTO TABLE players (col1, col2, col3...) FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n';