I looked for a similar question but I could not find a magic answer....so I'm hoping I'll find one now as this is driving me crazy!
I was going great guns with importing CSV data into MySQL...and now suddenly I'm getting nothing but skipped records. I ought to mention that I have been doing lots of deleting records (just experimenting really as I'm learning about MySQL and building my first database) so I've been chopping and changing the data including lots of copy/pasting in Excel, just in case any of this may be causing the trouble.
An example here, where I chopped the CSV down to just 2 records:
GENRE_NAME
Classic Rock
Electronica
The query to load the data:
LOAD DATA INFILE 'Music_All_2.csv' IGNORE INTO TABLE genres COLUMNS TERMINATED BY ',' ESCAPED BY '\\' LINES TERMINATED BY '\r' IGNORE 1 LINES (genre_name);
Query OK, 0 rows affected (0.00 sec)
Records: 2 Deleted: 0 Skipped: 2 Warnings: 0
And the table is empty (so it's not skipping them due to them already existing):
select * from genres;
Empty set (0.00 sec)
Finally, here's the SHOW CREATE TABLE genres output:
genres | CREATE TABLE `genres` (
`genre_pk` tinyint(2) unsigned NOT NULL AUTO_INCREMENT,
`genre_name` varchar(90) NOT NULL,
PRIMARY KEY (`genre_pk`),
UNIQUE KEY `genre_name` (`genre_name`)
) ENGINE=InnoDB AUTO_INCREMENT=255 DEFAULT CHARSET=latin1 |
Thank you in advance to the solver of my problem, there is sooooo much I don't know - it's fun learning but frustrating at the same time!
I think you have 2 problems.
1. AUTO_INCREMENT value
In your DDL,
genre_pk is TINYINT. So genre_pk can hold 0~255.
in end of CREATE STATEMENT, AUTO_INCREMENT=255
This sets initial value for genre_pk to 255
2. LINES TERMINATED BY
You need to change LINES TERMINATED BY from '\r' to '\n'
3. Summary
I've tested following SQL, and it worked well.
DDL
CREATE TABLE `genres` (
`genre_pk` tinyint(2) unsigned NOT NULL AUTO_INCREMENT,
`genre_name` varchar(90) NOT NULL,
PRIMARY KEY (`genre_pk`),
UNIQUE KEY `genre_name` (`genre_name`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1;
LOAD DATA
LOAD DATA INFILE '/path/to/Music_All_2.csv'
INTO TABLE genres FIELDS TERMINATED BY ',' ESCAPED BY '\\'
LINES TERMINATED BY '\n' IGNORE 1 LINES (genre_name);
Reply to OP's comment
1) How to reset AUTO_INCREMENTed value
When table has data which shouldn't be deleted.
ALTER TABLE table_name AUTO_INCREMENT = n;
When existing data could be deleted. (this is what you've already tried)
TRUNCATE TABLE table_name;
2) '\n' v.s '\r'
That's weird. Generally speaking,
'\n' for Unix system
'\r\n' for Windows system
I never heard about '\r' for carriage return. MySQL manaul(http://dev.mysql.com/doc/refman/5.5/en/load-data.html) states that
If you have generated the text file on a Windows system, you might have to use LINES TERMINATED BY '\r\n' to read the file properly, because Windows programs typically use two characters as a line terminator. Some programs, such as WordPad, might use \r as a line terminator when writing files. To read such files, use LINES TERMINATED BY '\r'.
Related
I have a .csv file which look like this:
and that can be downloaded from here.
I create my DB and my table with the following code:
CREATE DATABASE test_schema;
CREATE TABLE test_schema.teams (
teamkey SMALLINT NOT NULL AUTO_INCREMENT,
teamid CHAR(3) NOT NULL,
yearid YEAR(4) NOT NULL,
leagueid CHAR(2) NOT NULL,
teamrank TINYINT(2) NOT NULL,
PRIMARY KEY (teamkey),
UNIQUE KEY teamkey_UNIQUE (teamkey),
KEY teamid_yearid_leagueid_UNIQUE (teamid, yearid, leagueid),
CONSTRAINT check_teamrank CHECK (((teamrank >= 0) and (teamrank <= 12))),
CONSTRAINT check_year CHECK (((yearid >= 1871) and (yearid <=2155))))
ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
Now, when I try to import using:
LOAD DATA LOCAL INFILE "path_to_file_in_my_computer/Teams.csv"
INTO TABLE test_schema.teams
FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(#teamID, #yearID, #lgID, #Rank);
I get 2895 warnings which are all the same:
Warning (Code 3819): Check constraint 'check_year' is violated.
This warning makes no sense since yearid goes from 1871 to 2018 as can be corroborated if you look at the structure of the Teams.csv file. So any advice or suggestion on how to handle this error will be much appreciated. I'm working on MySQL Workbench 8.0.
PS: I posted a similar question (deleted) today morning but it needed more details that are provided here.
You don't have the column names in the correct order in the LOAD DATA query.
LOAD DATA LOCAL INFILE "path_to_file_in_my_computer/Teams.csv"
INTO TABLE test_schema.teams
FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(yearid, leagueid, teamid, #franchid, #divid, teamrank);
You can assign directly to the table column names, you don't need to use #yearID unless you have to do extra processing before storing in the table.
I have created a table that has the following definition:
CREATE TABLE vacayhome.photo (
id INT NOT NULL AUTO_INCREMENT,
url_path CHAR NOT NULL,
caption CHAR NOT NULL,
space_type CHAR NOT NULL,
is_main BOOLEAN NOT NULL,
listing_id INT NOT NULL,
PRIMARY KEY (id),
FOREIGN KEY (listing_id) REFERENCES listing (id)
ON UPDATE RESTRICT
);
The data I'm trying to insert to this table has the following form:
url_path,caption,space_type,is_main,listing_id
JFK/015ec48e93480.jpg,An interior designer dream,,true,10000000
JFK/9184bd57e9f80.jpg,"Ready for you, YASS",,false,10000000
BCN/5ccd9b138c76.jpg,"Stay -- you're welcome",,false,10000001
BCN/5fbb3a5ac2b30.jpg,Warm sunlight throughout,,false,10000001
And this is my LOAD DATA statement into the empty table from the MariaDB and MySQL documentation:
ALTER TABLE photo DISABLE KEYS;
BEGIN;
LOAD DATA INFILE '/path/to/photos.csv' INTO TABLE photo
FIELDS
OPTIONALLY ENCLOSED BY '"'
TERMINATED BY ','
LINES
TERMINATED BY '\n'
IGNORE 1 ROWS (url_path,caption,space_type,is_main,listing_id);
SET id=NULL;
COMMIT;
ALTER TABLE photo ENABLE KEYS;
The OPTIONALLY ENCLOSED BY is being used for the CHAR columns that have a comma in the text, and the IGNORE 1 ROWS is used to ignore the header row.
When I try to load the data, I get the following error:
ERROR 1193 (HY000) at line 33: Unknown system variable 'id'
I've also tried adding SET id = NULL from answers in other StackOverflow posts like this one. What am I doing wrong?
You have an extra semicolon before the SET ID=NULL; directive.
Below code ran successfully in my local MySQL environment:
ALTER TABLE photo DISABLE KEYS;
BEGIN;
LOAD DATA LOCAL INFILE '/path/to/photos.csv' INTO TABLE photo
FIELDS
OPTIONALLY ENCLOSED BY '"'
TERMINATED BY ','
LINES
TERMINATED BY '\n'
IGNORE 1 ROWS (url_path,caption,space_type,is_main,listing_id)
SET id=NULL;
COMMIT;
ALTER TABLE photo ENABLE KEYS;
You might want to check the column types as well (for example, use VARCHAR instead). Good luck.
Have you tried putting the path in quotes?
As shown below...
url_path,caption,space_type,is_main,listing_id
"JFK/9184bd57e9f80.jpg","Ready for you, YASS",,false,10000000
I try to import a CSV file in my database, I don't have any error but no rows are insert
I checked my field name
My table structure is:
CREATE TABLE IF NOT EXISTS `tmp` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`test` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
and my CSV looks like:
id;test
1;11
2;22
The command I use to import the file is
LOAD DATA LOCAL INFILE '.../test.csv' INTO TABLE tmp FIELDS TERMINATED BY ';' LINES TERMINATED BY '\n' IGNORE 1 ROWS
It should import data into the tmp table, but just return No error, 0 row inserted
It could be because you did not specify the column to add the data in or the command "enclosed by". You could try adding those 2 lines you your code as shown below:
LOAD DATA INFILE '/path/to/test.csv'
INTO TABLE tmp
FIELDS TERMINATED BY ';'
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(id,test);
I used this article's step as reference for importing csv, you could refer to it for additional detail and explanation:
https://blog.terresquall.com/2021/11/importing-a-csv-file-into-an-sql-table/
If your CSV is in a notepad like mine during testing, a "data truncated" warning will appear because there is a newline character at the end of each line. It can be ignored as it does not affect the results.
I have a CSV file with one column with the next data:
"2015-01-01",
...
...
"2015-03-27"
I created mysql table that way:
CREATE TABLE `my_tbl` ( `id` int(11) NOT NULL AUTO_INCREMENT, `date` DATE NOT NULL, PRIMARY KEY (`id`) );
I am trying to insert data using the next command:
LOAD DATA INFILE '/tmp/myFile.csv' INTO TABLE my_tbl FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (#col1) set date=#col1;
The problem: When checking my_tbl I see that all records are '0000-00-00'
What am I doing wrong and how can I fix it?
Did I define the table as it should be (e.g. Maybe it was better to define timestamp etc.)
Spot the difference:
[..snip..] BY '\n' (#col1) set date=#co1;
^---------------^
It appears I should have added FIELDS ENCLOSED BY '\"'. So the LOAD query should be this way:
LOAD DATA INFILE '/tmp/myFile.csv' INTO TABLE my_tbl FIELDS ENCLOSED BY '\"' LINES TERMINATED BY '\n' (#col1) set date=#col1;
Tnx for user #Marc B for the hint
I am trying to import a csv file that is delimited by tabs.
Here is my query
LOAD DATA LOCAL INFILE 'c:/news.csv'
INTO TABLE news
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\r'
(url, storyid, title, date, details, category, author);
What happens is only the first column is loaded, (url).
The rest shows NULL. I have tried lines terminated by \n as well. Same result.
Any advice?
Table structure for table `news`
--
CREATE TABLE IF NOT EXISTS `news` (
`url` varchar(62) DEFAULT NULL,
`storyid` int(15) DEFAULT NULL,
`title` varchar(255) DEFAULT NULL,
`date` date DEFAULT NULL,
`details` longtext,
`category` varchar(255) DEFAULT NULL,
`author` varchar(110) DEFAULT NULL
)
It depends on the exact format of your .csv file but for Windows .csv format I always use
LINES TERMINATED BY '\r\n'
also (again depending on the data) try
FIELDS ESCAPED BY '\\' TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '\"'
If you're unsure of the exact nature of the data sometimes it is better to view it in hexadecimal to see how the lines are really terminated. I use Hexedit - http://www.hexedit.com/
Hope this helps.
Dermot
Like I said in the comments you can use '/r/n' for a new line.
However your csv file contains only 1 column, namely a full line of text.
That is probably also why only the first table column is filled and the rest is null.
LOAD DATA LOCAL INFILE 'c:/news.csv'
INTO TABLE news
COLUMNS TERMINATED BY ','
LINES TERMINATED BY '\r\n'
(url, storyid, title, date, details, category, author)
This worked.
Turned out that even though it looks tab separated, it is comma separated. Dermot was right that you need to view it in hexadecimal view to see how it is really deliminated.