I'm trying to read a csv file and load the values to a mysql table.
My csv file looks like this:
"1026235","2172","Werdmühlestrasse","4","400","Werdmühlestrasse 4","3","real","BB","261AA01857","3169179","2683137.449","1247708.724","8001","0","Zürich","AA1750","","K","1","Lindenhof","13","1301","Zürichberg","Altstadt","St.Peter u Paul","St.Peter","1026238","562","Fortunagasse","15","1500","Fortunagasse 15","3","real","BB","261AA01852","140709","2683163.645","1247502.811","8001","0","Zürich","AA5297","","K","1","Lindenhof","13","1301","Zürichberg","Altstadt","St.Peter u Paul","St.Peter","1","3","3","29.0","8.539764579706915","47.373115180353350","POINT (2683163.8 1247502.8)"
This is the command I'm trying to run:
LOAD DATA INFILE '/home/coder/project/geoz.adrstzh_adressen_stzh_p.csv'
INTO TABLE mainZuerichAddresses
FIELDS
TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(#col1,#dummy,#col3,#col4,#dummy,#col6,#col7,#dummy,#dummy,#col10,#dummy,#dummy,#dummy,#col14,#dummy,#col16,#dummy,#dummy,#dummy,#col20,#col21,#dummy,#col23,#col24,#col25,#col26,#col27,#col28,#col29,#dummy,#dummy,#dummy,#dummy,#col34)
SET objid=#col29,gebaeudeeingangnummer=#col1,adresse=#col6,lokalisationsname=#col3,
hausnummer=#col4,plz=#col14,plz_ortschaft=#col16,stadtkreis=#col20,
gebaeudenummer=#col10,statistisches_quartier=#col21,status=#col7,statistische_zone=#col23,schulkreis=#col24,verwaltungsquartier=#col25,
roem_kath_kirchgemeinde=#col26,ev_ref_kirchgemeinde=#col27,ev_ref_kirchenkreis=#col28,geometry=#col34;
Here I added all the 34 columns from the csv file:
(#col1,#dummy,#col3,#col4,#dummy,#col6,#col7,#dummy,#dummy,#col10,#dummy,#dummy,#dummy,#col14,#dummy,#col16,#dummy,#dummy,#dummy,#col20,#col21,#dummy,#col23,#col24,#col25,#col26,#col27,#col28,#col29,#dummy,#dummy,#dummy,#dummy,#col34)
and here I'm trying to add the data to the table columns I have, which are in a different order than the csv and I don't need all of them, only 18. (Can I even do that, cherry-pick columns from the csv file and mix their order?)
SET objid=#col29,gebaeudeeingangnummer=#col1,adresse=#col6,lokalisationsname=#col3,hausnummer=#col4,plz=#col14,plz_ortschaft=#col16,stadtkreis=#col20,gebaeudenummer=#col10,statistisches_quartier=#col21,status=#col7,statistische_zone=#col23,schulkreis=#col24,verwaltungsquartier=#col25,roem_kath_kirchgemeinde=#col26,ev_ref_kirchgemeinde=#col27,ev_ref_kirchenkreis=#col28,geometry=#col34;
But I'm keep getting this error:
ERROR 1366 (HY000): Incorrect integer value: 'Werdmühlestrasse 4' for column 'plz' at row 1
I read the documentation, but it's not very clear how the mysql should be formatted.:
You must also specify a column list if the order of the fields in the
input file differs from the order of the columns in the table.
Otherwise, MySQL cannot tell how to match input fields with table
columns.
I based my mysql command on this question, but it's quite old.
I also found this question which gave some advice about FIELDS and LINES termination so I played around a bit with that.
I'm not sure if the csv formatting is the problem or the order I'm trying to load the data from the csv into the table colums.
Someone has an idea?
look carefully on error message.
There says value: 'Werdmühlestrasse 4' is not integer.
There are a number of questions in your question for this part 'it's not very clear how the mysql should be formatted'
What appears in brackets defines the order of columns in the csv file and should include all columns for example
given a csv file
name,junk,val
mike,1234,aaa
bob,4567,bbb
steve,8910,ccc
and a table
create table t(id int auto_increment primary key,
name varchar(20),
junk varchar(20),
val varchar(20));
The following will fail because I have not provided a column list in the csv file and load data infile attempts to load the first field in input file to id and the datatype does not match the datatype for id in the table.
LOAD DATA INFILE 'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 ROWS;
in fact I want to allow auto increment so I specify the target columns for all the input file columns.
LOAD DATA INFILE 'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 ROWS
(name,junk,val);
+----+-------+------+------+
| id | name | junk | val |
+----+-------+------+------+
| 1 | mike | 1234 | aaa |
| 2 | bob | 4567 | bbb |
| 3 | steve | 8910 | ccc |
+----+-------+------+------+
3 rows in set (0.001 sec)
and if I want col3 in the file for go to name in table and col1 in file to go to val in table
LOAD DATA INFILE 'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 ROWS
(val,junk,name);
+----+-------+------+-------+
| id | name | junk | val |
+----+-------+------+-------+
| 1 | mike | 1234 | aaa |
| 2 | bob | 4567 | bbb |
| 3 | steve | 8910 | ccc |
| 4 | aaa | 1234 | mike |
| 5 | bbb | 4567 | bob |
| 6 | ccc | 8910 | steve |
+----+-------+------+-------+
6 rows in set (0.001 sec)
and if I want to load a column from the input file park it in a user defined variable and do nothing with it (as you have done)
LOAD DATA INFILE 'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 ROWS
(val,#dummy,name);
+----+-------+------+-------+
| id | name | junk | val |
+----+-------+------+-------+
| 1 | mike | 1234 | aaa |
| 2 | bob | 4567 | bbb |
| 3 | steve | 8910 | ccc |
| 4 | aaa | 1234 | mike |
| 5 | bbb | 4567 | bob |
| 6 | ccc | 8910 | steve |
| 7 | aaa | NULL | mike |
| 8 | bbb | NULL | bob |
| 9 | ccc | NULL | steve |
+----+-------+------+-------+
9 rows in set (0.001 sec)
Another use for the user defined variables is for input processing see the manual for examples of this https://dev.mysql.com/doc/refman/8.0/en/load-data.html
in your case you aren't doing any input transformations so all those set statements appear to be unnecessary, but will work.
I have looked on previous forums for this, websites etc and can't find a solution. I keep getting this error despite me having 8 columns in my database and my csv file which I'm trying to load into the database.
I have included screenshots of my command line, database table which im loading into and my csv file.
Any help is much appreciated!
Any suggestions on this please??
It’s the line endings. MySQL isn’t getting what it expects, so specify the format of the file using LINES TERMINATED BY ‘\r\n’ or whatever is appropriate for you:
‘\r\n’ for files that came from Windows systems
‘\r’ for files from VMS
‘\n’ for every other source.
The issue is not related to the new lines but related to the commas within the data itself as seen in the "Full Name" column in your data.
Was able to replicate and fix the issue.
Database table used for the replication of the issue, please note that the data definitely fits within the column.
mysql> describe import;
+-------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| col1 | varchar(255) | YES | | NULL | |
| col2 | varchar(255) | YES | | NULL | |
+-------+--------------+------+-----+---------+-------+
2 rows in set (0.01 sec)
Replication:
I am sure that the line terminator is \n as the file was created on linux.
# cat /var/lib/mysql-files/import.csv
col1,col2
test1,value1,value2
test2,value3
SQL statement that gives the issue:
LOAD DATA INFILE '/var/lib/mysql-files/import.csv'
INTO TABLE import
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES;
Error:
ERROR 1262 (01000): Row 1 was truncated; it contained more data than there were input columns
Solution:
I had to change the data file and the SQL statement to make the issue go away.
I made sure that the data contained double quotes around the columns:
# cat /var/lib/mysql-files/import.csv
"col1","col2"
"test1","value1,value2"
"test2","value3"
Updated the SQL statement to know that the fields are enclosed by double quotes, seen "ENCLOSED BY '"'":
LOAD DATA INFILE '/var/lib/mysql-files/import.csv'
INTO TABLE import
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES;
Result:
Query OK, 2 rows affected (0.01 sec)
Records: 2 Deleted: 0 Skipped: 0 Warnings: 0
Import was successful:
mysql> select * from import;
+-------+---------------+
| col1 | col2 |
+-------+---------------+
| test1 | value1,value2 |
| test2 | value3 |
+-------+---------------+
2 rows in set (0.00 sec)
I would import one million rows from a CSV document to a database table.
To do this fast i use MySQL load data infile.
The problem is that everything works exactly great!
But there is a problem with lines which optionally enclosed by ".
The CSV file.
Name|Value\n
Anna|\n
Ben |Test\n
Chip|"My ""special"" value"\n
Deny|"I" like it\n
East|You not\n
The MySQL command.
LOAD DATA LOCAL INFILE 'test.csv'
INTO TABLE `test`
FIELDS TERMINATED BY '|'
ENCLOSED BY '"'
LINES TERMINATED BY "\n"
IGNORE 1 LINES
(`name`, #value)
SET
`value` = nullif(#value, '')
;
The result.
Query OK, 4 rows affected, 1 warning (0.17 sec)
Records: 4 Deleted: 0 Skipped: 0 Warnings: 1
The warnings.
+---------+------+--------------------------------------------+
| Level | Code | Message |
+---------+------+--------------------------------------------+
| Warning | 1265 | Data truncated for column 'value' at row 4 |
+---------+------+--------------------------------------------+
The table.
+----+------+------------------------+
| id | name | value |
+----+------+------------------------+
| 1 | Anna | NULL |
| 2 | Ben | Test |
| 3 | Chip | My "special" value |
| 4 | Deny | "I" like it
East|You |
+----+------+------------------------+
How to solve?
Please note:
My problem is not the warning!
If you see: The csv file contains 6 lines and 5 rows. (without header)
Also i need 5 rows/entries in mysql table. I have only 4 entries.
Your csv-file is invalid: Line 5 contains a "-char so (according to RFC-4180)
the field has to be surrounded by double quotes and
the quotes inside the field have to be repeatet
Using this you can successfully Import your csv by modifying it to
Name|Value
Anna|
Ben |Test
Chip|"My ""special"" value"
Deny|"""I"" like it"
East|You not
This Data truncated for column 'value' at row 4 warning message indicate that your field value is greater then your specified size. So you have to increase size of varchar to maximum length of your value.