I'm trying to read a csv file and load the values to a mysql table.
My csv file looks like this:
"1026235","2172","Werdmühlestrasse","4","400","Werdmühlestrasse 4","3","real","BB","261AA01857","3169179","2683137.449","1247708.724","8001","0","Zürich","AA1750","","K","1","Lindenhof","13","1301","Zürichberg","Altstadt","St.Peter u Paul","St.Peter","1026238","562","Fortunagasse","15","1500","Fortunagasse 15","3","real","BB","261AA01852","140709","2683163.645","1247502.811","8001","0","Zürich","AA5297","","K","1","Lindenhof","13","1301","Zürichberg","Altstadt","St.Peter u Paul","St.Peter","1","3","3","29.0","8.539764579706915","47.373115180353350","POINT (2683163.8 1247502.8)"
This is the command I'm trying to run:
LOAD DATA INFILE '/home/coder/project/geoz.adrstzh_adressen_stzh_p.csv'
INTO TABLE mainZuerichAddresses
FIELDS
TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(#col1,#dummy,#col3,#col4,#dummy,#col6,#col7,#dummy,#dummy,#col10,#dummy,#dummy,#dummy,#col14,#dummy,#col16,#dummy,#dummy,#dummy,#col20,#col21,#dummy,#col23,#col24,#col25,#col26,#col27,#col28,#col29,#dummy,#dummy,#dummy,#dummy,#col34)
SET objid=#col29,gebaeudeeingangnummer=#col1,adresse=#col6,lokalisationsname=#col3,
hausnummer=#col4,plz=#col14,plz_ortschaft=#col16,stadtkreis=#col20,
gebaeudenummer=#col10,statistisches_quartier=#col21,status=#col7,statistische_zone=#col23,schulkreis=#col24,verwaltungsquartier=#col25,
roem_kath_kirchgemeinde=#col26,ev_ref_kirchgemeinde=#col27,ev_ref_kirchenkreis=#col28,geometry=#col34;
Here I added all the 34 columns from the csv file:
(#col1,#dummy,#col3,#col4,#dummy,#col6,#col7,#dummy,#dummy,#col10,#dummy,#dummy,#dummy,#col14,#dummy,#col16,#dummy,#dummy,#dummy,#col20,#col21,#dummy,#col23,#col24,#col25,#col26,#col27,#col28,#col29,#dummy,#dummy,#dummy,#dummy,#col34)
and here I'm trying to add the data to the table columns I have, which are in a different order than the csv and I don't need all of them, only 18. (Can I even do that, cherry-pick columns from the csv file and mix their order?)
SET objid=#col29,gebaeudeeingangnummer=#col1,adresse=#col6,lokalisationsname=#col3,hausnummer=#col4,plz=#col14,plz_ortschaft=#col16,stadtkreis=#col20,gebaeudenummer=#col10,statistisches_quartier=#col21,status=#col7,statistische_zone=#col23,schulkreis=#col24,verwaltungsquartier=#col25,roem_kath_kirchgemeinde=#col26,ev_ref_kirchgemeinde=#col27,ev_ref_kirchenkreis=#col28,geometry=#col34;
But I'm keep getting this error:
ERROR 1366 (HY000): Incorrect integer value: 'Werdmühlestrasse 4' for column 'plz' at row 1
I read the documentation, but it's not very clear how the mysql should be formatted.:
You must also specify a column list if the order of the fields in the
input file differs from the order of the columns in the table.
Otherwise, MySQL cannot tell how to match input fields with table
columns.
I based my mysql command on this question, but it's quite old.
I also found this question which gave some advice about FIELDS and LINES termination so I played around a bit with that.
I'm not sure if the csv formatting is the problem or the order I'm trying to load the data from the csv into the table colums.
Someone has an idea?
look carefully on error message.
There says value: 'Werdmühlestrasse 4' is not integer.
There are a number of questions in your question for this part 'it's not very clear how the mysql should be formatted'
What appears in brackets defines the order of columns in the csv file and should include all columns for example
given a csv file
name,junk,val
mike,1234,aaa
bob,4567,bbb
steve,8910,ccc
and a table
create table t(id int auto_increment primary key,
name varchar(20),
junk varchar(20),
val varchar(20));
The following will fail because I have not provided a column list in the csv file and load data infile attempts to load the first field in input file to id and the datatype does not match the datatype for id in the table.
LOAD DATA INFILE 'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 ROWS;
in fact I want to allow auto increment so I specify the target columns for all the input file columns.
LOAD DATA INFILE 'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 ROWS
(name,junk,val);
+----+-------+------+------+
| id | name | junk | val |
+----+-------+------+------+
| 1 | mike | 1234 | aaa |
| 2 | bob | 4567 | bbb |
| 3 | steve | 8910 | ccc |
+----+-------+------+------+
3 rows in set (0.001 sec)
and if I want col3 in the file for go to name in table and col1 in file to go to val in table
LOAD DATA INFILE 'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 ROWS
(val,junk,name);
+----+-------+------+-------+
| id | name | junk | val |
+----+-------+------+-------+
| 1 | mike | 1234 | aaa |
| 2 | bob | 4567 | bbb |
| 3 | steve | 8910 | ccc |
| 4 | aaa | 1234 | mike |
| 5 | bbb | 4567 | bob |
| 6 | ccc | 8910 | steve |
+----+-------+------+-------+
6 rows in set (0.001 sec)
and if I want to load a column from the input file park it in a user defined variable and do nothing with it (as you have done)
LOAD DATA INFILE 'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 ROWS
(val,#dummy,name);
+----+-------+------+-------+
| id | name | junk | val |
+----+-------+------+-------+
| 1 | mike | 1234 | aaa |
| 2 | bob | 4567 | bbb |
| 3 | steve | 8910 | ccc |
| 4 | aaa | 1234 | mike |
| 5 | bbb | 4567 | bob |
| 6 | ccc | 8910 | steve |
| 7 | aaa | NULL | mike |
| 8 | bbb | NULL | bob |
| 9 | ccc | NULL | steve |
+----+-------+------+-------+
9 rows in set (0.001 sec)
Another use for the user defined variables is for input processing see the manual for examples of this https://dev.mysql.com/doc/refman/8.0/en/load-data.html
in your case you aren't doing any input transformations so all those set statements appear to be unnecessary, but will work.
I am importing some data from a CSV file into a MySQL db. This has been workign well until I had a row come up which uses a single quote in the Description field.
The problematic row in the CSV is this:
"2019","3982","2018-12-16","black, height - 16.0 ', matte, waterproof, ellipsoid","42","PXC Mfg Inc","emarkham#______.com"
The SQL import is
LOAD DATA INFILE :file
INTO TABLE $table_name
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(Year,Price,StockDate,Description,Quantity,Supplier,Contact)
Given that all columns are double quoted, I would expect the result to be
Year | Price | StockDate | Description | Quantity | Supplier | Contact
------+-------+------------+------------------------------------------------------+----------+-------------+---------------------
2019 | 3982 | 2018-12-16 | black, height - 16.0 ', matte, waterproof, ellipsoid | 42 | PXC Mfg Inc | emarkham#______.com
but what I actually got was
Year | Price | StockDate | Description | Quantity | Supplier | Contact
------+-------+------------+----------------------+-----------+-----------+----------
2019 | 3982 | 2018-12-16 | black, height - 16.0 | 0 | waterproof| ellipsoid
As you can see the single quote (indicating 16 inches) has been interpreted as the end of that value. This is certainly not the desired result.
Anyone know how to correct this?
Query:
LOAD DATA LOCAL INFILE 'actors.csv'
INTO TABLE Actors
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(ACTOR_ID, FNAME, LNAME);
CSV File:
ACTOR_ID, FNAME, LNAME
"66666","Billy","Lou"
"77777","Sally","Lou"
"88888","Hilly","Lou"
mysql> describe Actors;
+----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+
| ACTOR_ID | char(5) | NO | PRI | | |
| FNAME | varchar(20) | NO | | NULL | |
| LNAME | varchar(20) | NO | | NULL | |
+----------+-------------+------+-----+---------+-------+
> The output after running query:
| 10047 | Shirley | Jones |
| 10048 | Andre | Vippolis |
| 66666 | Billy | Lou"
"77777 |
| 88888 | Hilly | "Lou"
|
+----------+-------------+---------------+
I am trying to put a CSV file into my database. I've gotten the query
from a MySQL tutorial (except put the values I have in there). When I
run the query, My data is not properly inserted. I already have 2 rows
inserted (10047, 10048) and then I try to put the data from the CSV
file in, but it does not go in properly. It seems that the quotations
are not being read properly. But the statement ENCLOSED BY '"'
should handle the quotations. What am I doing wrong here?
It seems there is \r between
"Lou"
"77777"
and not \n
Use text editor to correct this.
Found a related so post
CSV files frequently have a carriage return/line feed as the line terminator. If the file was generated using Excel, for example, you will almost definitely have that.
A way to correct that is to modify your code as follows:
LOAD DATA LOCAL INFILE 'actors.csv'
INTO TABLE Actors
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(ACTOR_ID, FNAME, LNAME);
I do most of my CSV importing that way.
I had brought this up before earlier, but after doing some research, I realized I was looking in the wrong place. Here is the situation. I create this table:
CREATE TABLE PC_Contacts
(
POC VARCHAR(255) PRIMARY KEY NOT NULL,
Phone_1 VARCHAR(255),
Phone_2 VARCHAR(255)
);
I import a CSV file into MySQL which has the values for my table PC_Contacts:
USE Network
LOAD DATA INFILE 'C:\\ProgramData\\MySQL\\MySQL Server 5.7\\Uploads\\PC_Contacts.csv'
INTO Table PC_Contacts
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;
My output after importing looks like this:
+------------------+--------------+---------------+
| POC | Phone_1 | Phone_2 |
+------------------+--------------+---------------+
|April Wilson | 123-456-5000 | 123-456-5006
| | 123-456-2222 |
| | 123-456-5331 |
| | 123-456-7772 |
|Anton Watson | 123-456-1258 | 123-456-6005
|Elisa Kerring | 123-456-1075 | 123-456-4475
Now as you may recall, based on my code input, that POC is the PK. I had, in the original CSV file, a value for every line. However, as you see, anything that has no value on the right affects the left column's values. However, if I looked in the GUI and pulled up the table there, it showed the cell as populated with the value, so the data is there. If I were to put in xxx-xxx-xxxx, it would fix the issue:
+------------------+--------------+---------------+
| POC | Phone_1 | Phone_2 |
+------------------+--------------+---------------+
|April Wilson | 123-456-5000 | 123-456-5006
|Nicky Nite | 123-456-2222 | xxx-xxx-xxxx
|Nicole | 123-456-5331 | xxx-xxx-xxxx
|Becky | 123-456-7772 | xxx-xxx-xxxx
|Anton Watson | 123-456-1258 | 123-456-6005
|Elisa Kerring | 123-456-1075 | 123-456-4475
Obviously my intentions are so that I can see the value without having to apply special formatting in the command line. Is there a special SELECT command for that maybe?
Here is a link to a portion of the .CSV, as requested:
https://drive.google.com/file/d/0B0MMqHN75RpGdkZhcGp0SWtmams/view?usp=sharing
Your CSV file contains a carriage return with newline at the end of row, which breaks formatting. Use:
SELECT POC, Phone_1, REPLACE(Phone_2, '\r', '') AS Phone_2 FROM PC_Contacts;
Or change your import query as follows:
USE Network
LOAD DATA INFILE 'C:\\ProgramData\\MySQL\\MySQL Server 5.7\\Uploads\\PC_Contacts.csv'
INTO Table PC_Contacts
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS;
And use simple SELECT:
SELECT * FROM PC_Contacts;
In the table theGlobal from MySql Database I have the field theTime set on char 50 and the field theType set on char 1.
On table theGlobal import with LOAD DATA INFILE Syntax two different .csv files.
In the first .csv file I have this row:
"T","0:01"
"B","1:05"
The format of 0:01 and 1:05 is mm:ss
In the second .csv file I have this row:
"L","00:07:10"
"L","01:21:39"
The format of 00:07:10 and 01:21:39 is hh:mm:ss
The result of import in the table theGlobal transform the mm of first .csv on the hh and the ss of first .csv on the mm.
E.g:
+---------+----------+
| theType | theTime |
+---------+----------+
| B | 1:05:00 |
| T | 0:01:00 |
| L | 00:07:10 |
| L | 01:21:39 |
+---------+----------+
I need for all rows in the field theTime the format hh:mm:ss.
+---------+----------+
| theType | theTime |
+---------+----------+
| B | 00:01:05 |
| B | 00:00:01 |
| L | 00:07:10 |
| L | 01:21:39 |
+---------+----------+
How to resolve this?
Please help me, thank you so much in advance.
While loading the data, you can assign it to a variable first, then do whatever with the variable and load it in the actual column. In your case this would look something like this:
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(column1, #var1)
SET column2 = TIME(STR_TO_DATE(#var1, '%i:%S'));
Adjust the STR_TO_DATE() parameter as needed. Here's a table explaining it (it's for date_format() but it's the same for str_to_date()).
Oh, and store the data in a time column, not varchar.