I have a CSV file with format as below
nconst,primaryName,birthYear,deathYear,primaryProfession,knownForTitles
nm0000001,Fred,1899,1987,"soundtrack,actor,miscellaneous","tt0072308,tt0045537"
nm0000002,Lauren,1924,2014,"actress,soundtrack","tt0038355,tt0117057,tt0037382"
Some fields are enclosed in quotes ("") which is not an issue and also fields have a comma (,) as a delimiter.
I am using below command in mysql command line:
load data local infile '/home/ec2-user/sample.csv' into table movies.`sample` fields terminated by ',' enclosed by '"' lines terminated by '\n' ignore 1 lines;
which intern gives no error but data is inserted in the table in below wrong format:
**nm0000001** Fred 1899 1987 soundtrack,actor,miscellaneous tt0043044,tt0072308,tt0050419,tt0045537" **nm0000002**,Lauren Bacall,1924,2014,"actress,soundtrack
As we can clearly see , data from 2nd row appends in the first row
Thanks in advance
EDIT :
table definition :
CREATE TABLE `sample` (
`nconst` text,
`primaryName` text,
`birthYear` int(11) DEFAULT NULL,
`deathYear` text,
`primaryProfession` text,
`knownForTitles` text
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Related
Below is the code that I use to import CSV file to MySQL database. It works well to divide all the field and its record.
LOAD DATA INFILE 'file.csv'
INTO TABLE customer FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n'
(
ID, name, salary, address, status
);
However, when there is a VARCHAR or TEXT field which consist of comma (','), it works improperly. It is because I use FIELDS TERMINATED BY ',' that used to separate each field record.
So, for example, if a customer with salary 50,000 (double), it split the field normally. But, if the customer address is Java Road 15, Hong Kong (varchar/text), Java Road 15 will be saved in address field, while the Hong Kong will be saved to status field. This basically remove any record inside the status field. Any clue for this problem? Thanks in advance.
Are the fields enclosed by double quotes or something else? If so, you can add the "ENCLOSED BY" in your query.
FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\r\n';
"Enclosed by" specifies the character to identify the start and end of a field. In your case, field is enclosed by a double quote such as "Java Road 15, Hong Kong". It helps MYSQL to extract the field correctly even if there is a field delimiter in the field.
MYSQL manual: https://dev.mysql.com/doc/refman/5.7/en/load-data.html
LOAD DATA INFILE 'file.csv'
INTO TABLE customer FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
(
ID, name, salary, address, status
);
Try this one.
If anyone happens to stumble upon this answer, I wanted to share b/c this took me way too long to figure out - using mac, datagrip, mysql 5.7+, survey response questions:
DROP TABLE IF EXISTS surveyQuestion_ID;
#create table
CREATE TABLE surveyQuestion_ID
(
surveyQuestion_ID INT(11) NOT NULL,
surveyDescription TEXT,
surveyResponse VARCHAR(25) DEFAULT NULL,
PRIMARY KEY (surveyQuestion_ID)
);
#load query
LOAD DATA LOCAL INFILE 'file_location/file.csv' INTO TABLE surveyQuestion_ID
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(surveyQuestion_ID, surveyDescription, surveyResponse);
Hope this helps.
I have a CSV file. How can I tell the sqlldr control file to load missing values as NULL. (ie the table schema allows NULL for certain column)
Example of CSV
1,Name1
2,Name2
3,
4,Name3
Could you help me to edit my control file here so that a line 3 , the missing value is inserted as NULL in my table
Table
Create table test
( id Number(2), name Varchar(10), primary key (id) );
Control file
LOAD DATA INFILE '{path}\CSVfile.txt'
INSERT INTO test
FIELDS TERMINATED BY ','
(id CHAR,
name CHAR
)
I believe all you should have to do is this:
name CHAR(10) NULLIF(name=BLANKS)
You would have to hint to SQL*Loader that there might be nulls in your data.
2 ways to give that hint to SQL*Loader.
Use TRAILING NULLCOLS option.
LOAD DATA INFILE '{path}\CSVfile.txt'
INSERT INTO test<br>
FIELDS TERMINATED BY ','
TRAILING NULLCOLS
(id CHAR,
name CHAR
)
Recreate your CSV files with enclosed fields and then use OPTIONALLY ENCLOSED BY '"' which lets SQL*Loader clearly see the nulls in your data (nothing between quotes) that looks like "abcd",""
LOAD DATA INFILE '{path}\CSVfile.txt'
INSERT INTO test<br>
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
(id CHAR,
name CHAR
)
I found that using TRAILING NULLCOLS will do the job BUT it has to be for "blanks" at the end of the record line.
LOAD DATA INFILE {path}\Your_File
INSERT INTO TABLE Your_Table
TRAILING NULLCOLS
FIELDS TERMINATED BY ","
(
... your fields
)
Im importing a csv file to a mysql table with the following query;
"LOAD DATA INFILE 'myfielname.csv'
INTO table customers
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\r'
IGNORE 3 LINES
(sales,regional,accounts)
";
Is there any way to insert a string of characters before a field that is to be imported?
For example: The field 'sales' above refers to account id numbers, which are being used in the application. Id like to append a URL before account number during import so the final record in the table will be as follows:
String I want to come before 'sales', but within the same record: http://www.url.com?id=
If a given sales id was 1234 the final record in the table would be http://www.url.com?id=1234
Thanks in advance for your help.
Try someting like this
LOAD DATA LOCAL INFILE 'C:/test.csv'
INTO TABLE test.test1
FIELDS TERMINATED BY ';'
(#test1col,#test2col)
set test1col=CONCAT('http://url.com?id=',#test1col),test2col=#test2col;
The test csv has 2 columns. I created a test table like this
CREATE TABLE `test1` (
`test1col` varchar(200) DEFAULT NULL,
`test2col` varchar(2000) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
You could try immediatley with your own, just make sure you name the columns correctly!
Give it a try it worked for me.
I want to insert data into mysql table from csv file. Import data from region_codes.csv file. In region_codes.csv file having 3 columns in 3rd columns it had , separated data, include those commas how to insert in mysql.
DROP TABLE IF EXISTS `region_codes`;
CREATE TABLE `region_codes` (
`country_code` CHAR(2) NULL,
`region_no` varchar(5) NOT NULL,
`region` VARCHAR(45) NULL,
INDEX `idx_country_code` (`country_code`)
) COLLATE='utf8_bin' ENGINE = MyISAM;
Using LOAD DATA LOCAL INFILE I import the data but only 1000 rows are imported outof 4066 rows.
LOAD DATA LOCAL INFILE 'C:/region_codes.csv' INTO TABLE `region_codes` FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n';
How to insert huge amount of data into mysql region_codes table from csv file.
Screenshot:
You can try below syntax if it works for you otherwise provide csv data:
LOAD DATA LOCAL INFILE 'C:/region_codes.csv' INTO TABLE `region_codes` FIELDS ESCAPED BY '\\' FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n';
If above syntax does not work then export data by below command again and import by below given command.
select * into outfile 'C:/region_codes.csv' fields terminated by ',' optionally enclosed by '"' lines terminated by '\n' from `region_codes`;
Now use below command (to ignore column heading line)
LOAD DATA LOCAL INFILE 'C:/region_codes.csv' INTO TABLE `region_codes` FIELDS ESCAPED BY '\\' FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n' IGNORE 1 LINES;
Note: If data is prepared manually then need to correct it manually.
If still not work then attach your csv data to check exact problem.
You can use mysqlimport tool this way:
mysqlimport --ignore-lines=1 --fields-terminated-by=,
--columns='ID,Name,Phone,Address' --local -u root -p
Database /path/to/csvfile/TableName.csv
Here is a full explanation: http://chriseiffel.com/everything-linux/how-to-import-a-large-csv-file-to-mysql/
This is the table i want to import in:
create table if not exists Medici(
m_id int unsigned AUTO_INCREMENT PRIMARY KEY,
m_nume VARCHAR(50),
m_prenume VARCHAR(50),
Statut ENUM('primar', 'specialist'),
Specialitate VARCHAR(50)
) ENGINE = InnoDB DEFAULT CHARSET = utf8;
And the CSV starts with:
Nume,Prenume,Tip,Specialitate
Bunica,Mihai-Daniel,primar,reumatologie
Donca,Cornelia-Ana,primar,chirurgie
Achiriloaie,Lorand-Levente,specialist,neurologie
The code I wrote is:
load data infile 'D:/xxxxxxxxxxxxx/xxxxxxx/xxxxxxxxxx/xxxxxx/xxxxxxxxxx/medici.txt'
into table Medici
character set utf8
fields terminated by ','
lines starting by 'Nume,Prenume,Tip,Specialitate\n'
terminated by '\n'
(m_nume,m_prenume,Statut,Specialitate);
Note: I changed terminated with \n, \r, \r\n, and i still get 1 row. Even with removing lines terminated by and it's not working.
Note: I pasted quickly and messed up a bit, there are 4 rows there. And ye the csv i assume it's formated correctly
Can't wait for advice, Thanks in advance.
more of the code here:
Nume,Prenume,Tip,Specialitate
Bunica,Mihai-Daniel,primar,reumatologie
Donca,Cornelia-Ana,primar,chirurgie
Achiriloaie,Lorand-Levente,specialist,neurologie
Papuc,Raducu-Liviu,primar,homeopatie
Cucuiu,Nutu,primar,ortopedie
Buia,Tache,specialist,ginecologie
Dragomanu,Mitrut,specialist,ecografie
Ticu,Simona,specialist,psihiatrie
Ene,Adrian-Stefan,specialist,pediatrie
Copae,Toma,primar,neurologie
Hotoi,Dragos Alin,specialist,pediatrie
Ceafalau,Vincenţiu Mihail,primar,pediatrie
Briceag,Anca Stefana,primar,imagistica
Condrea,Nutu,primar,fizioterapie
Cruceru,Ioana-Loredana,primar,dermatologie
Soarece,Dan-Cristian,primar,o.r.l.
Tatasel,Alexandru-Ovidiu,specialist,psihologie
Sterian,Gologaneanu,primar,chirurgie
Postelnicu,Habib,primar,chirurgie
Silviu ,Adrian Ionut,primar,dermatologie
Paius,Ioana,specialist,ortopedie
Borza,Marius Florian,specialist,fizioterapie
Tamas,Ciprian Costel,primar,chirurgie
Ograzeanu,Cristina Alexandra,primar,endocrinologie
Rildo,Alex,specialist,ecografie
In the csv these lines are merged ( one row to another )
For example:
After reumatologie it starts with the name but when i paste the code you can see it's actually an \n
Bunica,Mihai-Daniel,primar,reumatologieDonca,(.... here is 2nd row and so on)
Try this command:
load data local infile
'D:/xxxxxxxxxxxxx/xxxxxxx/xxxxxxxxxx/xxxxxx/xxxxxxxxxx/medici.txt'
into table Medici character set utf8 fields terminated by ','
lines terminated by '\n' IGNORE 1 LINES (m_nume,m_prenume,Statut,Specialitate);