I'm trying to read a csv file and load the values to a mysql table.
My csv file looks like this:
"1026235","2172","Werdmühlestrasse","4","400","Werdmühlestrasse 4","3","real","BB","261AA01857","3169179","2683137.449","1247708.724","8001","0","Zürich","AA1750","","K","1","Lindenhof","13","1301","Zürichberg","Altstadt","St.Peter u Paul","St.Peter","1026238","562","Fortunagasse","15","1500","Fortunagasse 15","3","real","BB","261AA01852","140709","2683163.645","1247502.811","8001","0","Zürich","AA5297","","K","1","Lindenhof","13","1301","Zürichberg","Altstadt","St.Peter u Paul","St.Peter","1","3","3","29.0","8.539764579706915","47.373115180353350","POINT (2683163.8 1247502.8)"
This is the command I'm trying to run:
LOAD DATA INFILE '/home/coder/project/geoz.adrstzh_adressen_stzh_p.csv'
INTO TABLE mainZuerichAddresses
FIELDS
TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(#col1,#dummy,#col3,#col4,#dummy,#col6,#col7,#dummy,#dummy,#col10,#dummy,#dummy,#dummy,#col14,#dummy,#col16,#dummy,#dummy,#dummy,#col20,#col21,#dummy,#col23,#col24,#col25,#col26,#col27,#col28,#col29,#dummy,#dummy,#dummy,#dummy,#col34)
SET objid=#col29,gebaeudeeingangnummer=#col1,adresse=#col6,lokalisationsname=#col3,
hausnummer=#col4,plz=#col14,plz_ortschaft=#col16,stadtkreis=#col20,
gebaeudenummer=#col10,statistisches_quartier=#col21,status=#col7,statistische_zone=#col23,schulkreis=#col24,verwaltungsquartier=#col25,
roem_kath_kirchgemeinde=#col26,ev_ref_kirchgemeinde=#col27,ev_ref_kirchenkreis=#col28,geometry=#col34;
Here I added all the 34 columns from the csv file:
(#col1,#dummy,#col3,#col4,#dummy,#col6,#col7,#dummy,#dummy,#col10,#dummy,#dummy,#dummy,#col14,#dummy,#col16,#dummy,#dummy,#dummy,#col20,#col21,#dummy,#col23,#col24,#col25,#col26,#col27,#col28,#col29,#dummy,#dummy,#dummy,#dummy,#col34)
and here I'm trying to add the data to the table columns I have, which are in a different order than the csv and I don't need all of them, only 18. (Can I even do that, cherry-pick columns from the csv file and mix their order?)
SET objid=#col29,gebaeudeeingangnummer=#col1,adresse=#col6,lokalisationsname=#col3,hausnummer=#col4,plz=#col14,plz_ortschaft=#col16,stadtkreis=#col20,gebaeudenummer=#col10,statistisches_quartier=#col21,status=#col7,statistische_zone=#col23,schulkreis=#col24,verwaltungsquartier=#col25,roem_kath_kirchgemeinde=#col26,ev_ref_kirchgemeinde=#col27,ev_ref_kirchenkreis=#col28,geometry=#col34;
But I'm keep getting this error:
ERROR 1366 (HY000): Incorrect integer value: 'Werdmühlestrasse 4' for column 'plz' at row 1
I read the documentation, but it's not very clear how the mysql should be formatted.:
You must also specify a column list if the order of the fields in the
input file differs from the order of the columns in the table.
Otherwise, MySQL cannot tell how to match input fields with table
columns.
I based my mysql command on this question, but it's quite old.
I also found this question which gave some advice about FIELDS and LINES termination so I played around a bit with that.
I'm not sure if the csv formatting is the problem or the order I'm trying to load the data from the csv into the table colums.
Someone has an idea?
look carefully on error message.
There says value: 'Werdmühlestrasse 4' is not integer.
There are a number of questions in your question for this part 'it's not very clear how the mysql should be formatted'
What appears in brackets defines the order of columns in the csv file and should include all columns for example
given a csv file
name,junk,val
mike,1234,aaa
bob,4567,bbb
steve,8910,ccc
and a table
create table t(id int auto_increment primary key,
name varchar(20),
junk varchar(20),
val varchar(20));
The following will fail because I have not provided a column list in the csv file and load data infile attempts to load the first field in input file to id and the datatype does not match the datatype for id in the table.
LOAD DATA INFILE 'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 ROWS;
in fact I want to allow auto increment so I specify the target columns for all the input file columns.
LOAD DATA INFILE 'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 ROWS
(name,junk,val);
+----+-------+------+------+
| id | name | junk | val |
+----+-------+------+------+
| 1 | mike | 1234 | aaa |
| 2 | bob | 4567 | bbb |
| 3 | steve | 8910 | ccc |
+----+-------+------+------+
3 rows in set (0.001 sec)
and if I want col3 in the file for go to name in table and col1 in file to go to val in table
LOAD DATA INFILE 'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 ROWS
(val,junk,name);
+----+-------+------+-------+
| id | name | junk | val |
+----+-------+------+-------+
| 1 | mike | 1234 | aaa |
| 2 | bob | 4567 | bbb |
| 3 | steve | 8910 | ccc |
| 4 | aaa | 1234 | mike |
| 5 | bbb | 4567 | bob |
| 6 | ccc | 8910 | steve |
+----+-------+------+-------+
6 rows in set (0.001 sec)
and if I want to load a column from the input file park it in a user defined variable and do nothing with it (as you have done)
LOAD DATA INFILE 'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 ROWS
(val,#dummy,name);
+----+-------+------+-------+
| id | name | junk | val |
+----+-------+------+-------+
| 1 | mike | 1234 | aaa |
| 2 | bob | 4567 | bbb |
| 3 | steve | 8910 | ccc |
| 4 | aaa | 1234 | mike |
| 5 | bbb | 4567 | bob |
| 6 | ccc | 8910 | steve |
| 7 | aaa | NULL | mike |
| 8 | bbb | NULL | bob |
| 9 | ccc | NULL | steve |
+----+-------+------+-------+
9 rows in set (0.001 sec)
Another use for the user defined variables is for input processing see the manual for examples of this https://dev.mysql.com/doc/refman/8.0/en/load-data.html
in your case you aren't doing any input transformations so all those set statements appear to be unnecessary, but will work.
When I try to import my table named "Table.csv" to a MYSQl database using the linux console, it shows me the following message, ERROR 1265 (01000): Data truncated for column 'factor' at row 1
This is the description of the contents of the table that I had previously created:
+---------------------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------------+---------+------+-----+---------+-------+
| degrees | int(11) | YES | | NULL | |
| percentage | double | YES | | NULL | |
| factor | double | YES | | NULL | |
+---------------------+---------+------+-----+---------+-------+
This is how the data delimited only by commas "," is displayed as it is displayed from a plain text file:
1,0.35,1
2,0.10,3.0787
1,0.55,4.32
This is the sentence that I currently enter:
LOAD DATA INFILE '/var/lib/mysql-files/tabla.csv' INTO TABLE Student FIELDS
TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n';
In this case, there should be no problem because the table starts with the values from its header, since I have omitted the IGNORE 1 LINES statement;
I enclose the information where I check more questions:
https://dev.mysql.com/doc/refman/8.0/en/load-data.html
After verifying the way in which the data are structured and how the CSV file is stored, the sentence that should be used corresponds to:
LOAD DATA LOCAL INFILE '/var/lib/mysql-files/table.csv' INTO TABLE Student FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\ n';
Care should be taken with the type of data, its length and if it has some content, in this case all fields are filled with NULL, so they must be replaced by NOT NULL fields.
Query:
LOAD DATA LOCAL INFILE 'actors.csv'
INTO TABLE Actors
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(ACTOR_ID, FNAME, LNAME);
CSV File:
ACTOR_ID, FNAME, LNAME
"66666","Billy","Lou"
"77777","Sally","Lou"
"88888","Hilly","Lou"
mysql> describe Actors;
+----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+
| ACTOR_ID | char(5) | NO | PRI | | |
| FNAME | varchar(20) | NO | | NULL | |
| LNAME | varchar(20) | NO | | NULL | |
+----------+-------------+------+-----+---------+-------+
> The output after running query:
| 10047 | Shirley | Jones |
| 10048 | Andre | Vippolis |
| 66666 | Billy | Lou"
"77777 |
| 88888 | Hilly | "Lou"
|
+----------+-------------+---------------+
I am trying to put a CSV file into my database. I've gotten the query
from a MySQL tutorial (except put the values I have in there). When I
run the query, My data is not properly inserted. I already have 2 rows
inserted (10047, 10048) and then I try to put the data from the CSV
file in, but it does not go in properly. It seems that the quotations
are not being read properly. But the statement ENCLOSED BY '"'
should handle the quotations. What am I doing wrong here?
It seems there is \r between
"Lou"
"77777"
and not \n
Use text editor to correct this.
Found a related so post
CSV files frequently have a carriage return/line feed as the line terminator. If the file was generated using Excel, for example, you will almost definitely have that.
A way to correct that is to modify your code as follows:
LOAD DATA LOCAL INFILE 'actors.csv'
INTO TABLE Actors
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(ACTOR_ID, FNAME, LNAME);
I do most of my CSV importing that way.
I had brought this up before earlier, but after doing some research, I realized I was looking in the wrong place. Here is the situation. I create this table:
CREATE TABLE PC_Contacts
(
POC VARCHAR(255) PRIMARY KEY NOT NULL,
Phone_1 VARCHAR(255),
Phone_2 VARCHAR(255)
);
I import a CSV file into MySQL which has the values for my table PC_Contacts:
USE Network
LOAD DATA INFILE 'C:\\ProgramData\\MySQL\\MySQL Server 5.7\\Uploads\\PC_Contacts.csv'
INTO Table PC_Contacts
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;
My output after importing looks like this:
+------------------+--------------+---------------+
| POC | Phone_1 | Phone_2 |
+------------------+--------------+---------------+
|April Wilson | 123-456-5000 | 123-456-5006
| | 123-456-2222 |
| | 123-456-5331 |
| | 123-456-7772 |
|Anton Watson | 123-456-1258 | 123-456-6005
|Elisa Kerring | 123-456-1075 | 123-456-4475
Now as you may recall, based on my code input, that POC is the PK. I had, in the original CSV file, a value for every line. However, as you see, anything that has no value on the right affects the left column's values. However, if I looked in the GUI and pulled up the table there, it showed the cell as populated with the value, so the data is there. If I were to put in xxx-xxx-xxxx, it would fix the issue:
+------------------+--------------+---------------+
| POC | Phone_1 | Phone_2 |
+------------------+--------------+---------------+
|April Wilson | 123-456-5000 | 123-456-5006
|Nicky Nite | 123-456-2222 | xxx-xxx-xxxx
|Nicole | 123-456-5331 | xxx-xxx-xxxx
|Becky | 123-456-7772 | xxx-xxx-xxxx
|Anton Watson | 123-456-1258 | 123-456-6005
|Elisa Kerring | 123-456-1075 | 123-456-4475
Obviously my intentions are so that I can see the value without having to apply special formatting in the command line. Is there a special SELECT command for that maybe?
Here is a link to a portion of the .CSV, as requested:
https://drive.google.com/file/d/0B0MMqHN75RpGdkZhcGp0SWtmams/view?usp=sharing
Your CSV file contains a carriage return with newline at the end of row, which breaks formatting. Use:
SELECT POC, Phone_1, REPLACE(Phone_2, '\r', '') AS Phone_2 FROM PC_Contacts;
Or change your import query as follows:
USE Network
LOAD DATA INFILE 'C:\\ProgramData\\MySQL\\MySQL Server 5.7\\Uploads\\PC_Contacts.csv'
INTO Table PC_Contacts
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS;
And use simple SELECT:
SELECT * FROM PC_Contacts;