Tell sqlldr control file to load missing values as NULL - csv

I have a CSV file. How can I tell the sqlldr control file to load missing values as NULL. (ie the table schema allows NULL for certain column)
Example of CSV
1,Name1
2,Name2
3,
4,Name3
Could you help me to edit my control file here so that a line 3 , the missing value is inserted as NULL in my table
Table
Create table test
( id Number(2), name Varchar(10), primary key (id) );
Control file
LOAD DATA INFILE '{path}\CSVfile.txt'
INSERT INTO test
FIELDS TERMINATED BY ','
(id CHAR,
name CHAR
)

I believe all you should have to do is this:
name CHAR(10) NULLIF(name=BLANKS)

You would have to hint to SQL*Loader that there might be nulls in your data.
2 ways to give that hint to SQL*Loader.
Use TRAILING NULLCOLS option.
LOAD DATA INFILE '{path}\CSVfile.txt'
INSERT INTO test<br>
FIELDS TERMINATED BY ','
TRAILING NULLCOLS
(id CHAR,
name CHAR
)
Recreate your CSV files with enclosed fields and then use OPTIONALLY ENCLOSED BY '"' which lets SQL*Loader clearly see the nulls in your data (nothing between quotes) that looks like "abcd",""
LOAD DATA INFILE '{path}\CSVfile.txt'
INSERT INTO test<br>
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
(id CHAR,
name CHAR
)

I found that using TRAILING NULLCOLS will do the job BUT it has to be for "blanks" at the end of the record line.
LOAD DATA INFILE {path}\Your_File
INSERT INTO TABLE Your_Table
TRAILING NULLCOLS
FIELDS TERMINATED BY ","
(
... your fields
)

Related

mysql table formatting and infile issue

Each time I go and create my table and then add the .csv file, it keeps not fully loading in all of the data. Below is my code:
CREATE TABLE Parts_Maintenance (
Vehicle_ID BIGINT(20),
State VARCHAR(255),
Repair VARCHAR(255),
Reason VARCHAR(255),
Year YEAR,
Make VARCHAR(255),
Body_Type VARCHAR(255),
PRIMARY KEY (Vehicle_ID)
);
LOAD DATA INFILE '/home/codio/workspace/FleetMaintenanceRecords.csv'
INTO TABLE Parts_Maintenance
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\\n'
IGNORE 1 ROWS;
SELECT * FROM Parts_Maintenance;
Here is a photo of what it looks like in Codio:
And here is a photo of some of the data being brought in:
Could someone please help me pinpoint what I am doing wrong?
Tried to create table and bring in a .csv file. Table was created but the data is not all there and the table looks messed up
I agree with #barmer Your LINES TERMINATED BY is probably wrong, it's probably \r
You May use
LOAD DATA INFILE '/home/codio/workspace/FleetMaintenanceRecords.csv'
INTO TABLE Parts_Maintenance
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS;
Alternatively, you can use another method with workbench.
Right Click on Table
- Table data import wizard
- Choose the file path
Select the destination table
- Check and configure import settings
- Click on Next to execute.
This is the best and simple solution.

Import CSV with comma in a field with load data infile

I have a CSV file with format as below
nconst,primaryName,birthYear,deathYear,primaryProfession,knownForTitles
nm0000001,Fred,1899,1987,"soundtrack,actor,miscellaneous","tt0072308,tt0045537"
nm0000002,Lauren,1924,2014,"actress,soundtrack","tt0038355,tt0117057,tt0037382"
Some fields are enclosed in quotes ("") which is not an issue and also fields have a comma (,) as a delimiter.
I am using below command in mysql command line:
load data local infile '/home/ec2-user/sample.csv' into table movies.`sample` fields terminated by ',' enclosed by '"' lines terminated by '\n' ignore 1 lines;
which intern gives no error but data is inserted in the table in below wrong format:
**nm0000001** Fred 1899 1987 soundtrack,actor,miscellaneous tt0043044,tt0072308,tt0050419,tt0045537" **nm0000002**,Lauren Bacall,1924,2014,"actress,soundtrack
As we can clearly see , data from 2nd row appends in the first row
Thanks in advance
EDIT :
table definition :
CREATE TABLE `sample` (
`nconst` text,
`primaryName` text,
`birthYear` int(11) DEFAULT NULL,
`deathYear` text,
`primaryProfession` text,
`knownForTitles` text
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

MySQL: Load Data Infile from CSV that consist of comma in VARCHAR field

Below is the code that I use to import CSV file to MySQL database. It works well to divide all the field and its record.
LOAD DATA INFILE 'file.csv'
INTO TABLE customer FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n'
(
ID, name, salary, address, status
);
However, when there is a VARCHAR or TEXT field which consist of comma (','), it works improperly. It is because I use FIELDS TERMINATED BY ',' that used to separate each field record.
So, for example, if a customer with salary 50,000 (double), it split the field normally. But, if the customer address is Java Road 15, Hong Kong (varchar/text), Java Road 15 will be saved in address field, while the Hong Kong will be saved to status field. This basically remove any record inside the status field. Any clue for this problem? Thanks in advance.
Are the fields enclosed by double quotes or something else? If so, you can add the "ENCLOSED BY" in your query.
FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\r\n';
"Enclosed by" specifies the character to identify the start and end of a field. In your case, field is enclosed by a double quote such as "Java Road 15, Hong Kong". It helps MYSQL to extract the field correctly even if there is a field delimiter in the field.
MYSQL manual: https://dev.mysql.com/doc/refman/5.7/en/load-data.html
LOAD DATA INFILE 'file.csv'
INTO TABLE customer FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
(
ID, name, salary, address, status
);
Try this one.
If anyone happens to stumble upon this answer, I wanted to share b/c this took me way too long to figure out - using mac, datagrip, mysql 5.7+, survey response questions:
DROP TABLE IF EXISTS surveyQuestion_ID;
#create table
CREATE TABLE surveyQuestion_ID
(
surveyQuestion_ID INT(11) NOT NULL,
surveyDescription TEXT,
surveyResponse VARCHAR(25) DEFAULT NULL,
PRIMARY KEY (surveyQuestion_ID)
);
#load query
LOAD DATA LOCAL INFILE 'file_location/file.csv' INTO TABLE surveyQuestion_ID
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(surveyQuestion_ID, surveyDescription, surveyResponse);
Hope this helps.

ignore first two characters on a column while importing csv to mysql

I am trying to import a csv file to mysql table, But I need to remove First two characters on particular column before importing to mysql.
This is my statment :
string strLoadData = "LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets FIELDS terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES (SiteId,DateTime,Serial,DeviceId,AgentAID,VehicleRegistration,CarPark,SpaceNumber,GpsAddress,VehicleType,VehicleMake,VehicleModel,VehicleColour,IssueReasonCode,IssueReason,NoticeLocation,Points,Notes)";
Column IssueReasoncode' has data like 'LU12' , But i need to remove the first 2 characters it should have only integers on it and not alpha numeric .
I need to remove 'LU' from that column.
Is it possible to write like this on left(IssueReasonCode +' '2). This column is varchar(45) and cant be changed now because of large data on it.
Thanks
LOAD DATA INFILE has the ability to perform a function on the data for each column as you read it in (q.v. here). In your case, if you wanted to remove the first two characters from the IssueReasonCode column, you could use:
RIGHT(IssueReasonCode, CHAR_LENGTH(IssueReasonCode) - 2)
to remove the first two characters. You specify such column mappings at the end of the LOAD DATA statement using SET. Your statement should look something like the following:
LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets
FIELDS terminated by ','
ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(SiteId, DateTime, Serial, DeviceId, AgentAID, VehicleRegistration, CarPark, SpaceNumber,
GpsAddress, VehicleType, VehicleMake, VehicleModel, VehicleColour, IssueReasonCode,
IssueReason, NoticeLocation, Points, Notes)
SET IssueReasonCode = RIGHT(IssueReasonCode, CHAR_LENGTH(IssueReasonCode) - 2)
Referencing this and quoting this example , you can try the below to see if it works
User variables in the SET clause can be used in several ways. The
following example uses the first input column directly for the value
of t1.column1, and assigns the second input column to a user variable
that is subjected to a division operation before being used for the
value of t1.column2:
LOAD DATA INFILE 'file.txt' INTO TABLE t1 (column1, #var1) SET
column2 = #var1/100;
string strLoadData = "LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets FIELDS terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES (SiteId,DateTime,Serial,DeviceId,AgentAID,VehicleRegistration,CarPark,SpaceNumber,GpsAddress,VehicleType,VehicleMake,VehicleModel,VehicleColour,#IRC,IssueReason,NoticeLocation,Points,Notes) SET IssueReasonCode = substr(#IRC,2) ;";

load data terminated by does not import

Table:
ID int not null primary key auto_increment
number int not null unique
my file:
111
222
333
LOAD DATA LOCAL INFILE 'file.txt' IGNORE INTO TABLE MyTable fields terminated by '\n' (number); - everything works fine. But if I have:
file:
111;222;333
LOAD DATA LOCAL INFILE 'file.txt' IGNORE INTO TABLE MyTable fields terminated by ';' (number); - it imports only 111 and stops. Why?
If you want to add a list of values separated by a ";" into a single table field use this. This will basically treat each value as a separate record.
LOAD DATA LOCAL INFILE 'file.txt' IGNORE
INTO TABLE MyTable
LINES TERMINATED BY ';'
(number)
;
If you want to insert the 3 fields from the file into the first three fields in the table, remove (number). By including (number) you were specifying that you only wanted to insert data only into the number field.
LOAD DATA LOCAL INFILE 'file.txt' IGNORE
INTO TABLE MyTable fields terminated by ';';
If you want to insert the three fields from the file into three specific fields in the table (not necessarily the first three) you need to list all three. For example if you wanted to insert them into the fields number, field2 and field3 the command would be:
LOAD DATA LOCAL INFILE 'file.txt' IGNORE
INTO TABLE MyTable fields terminated by ';' (number, field2, field3);