I have a requirement to load data from a CSV file into a Mysql Table in Windows 10.
The command I am using in MySQL Workbench 8.0.20 is:
LOAD DATA INFILE "C:/ProgramData/MySQL/MySQL Server 5.7/Uploads/Sample.txt" IGNORE INTO TABLE TableA
FIELDS OPTIONALLY ENCLOSED BY '"'
TERMINATED BY ','
LINES TERMINATED BY '\r\n'
(col1, col2, col3, col4, col5, col6, col7, col8,
col9, col10, col11, col12, col13, col14, col15,
dummy_1, dummy_2, dummy_3, dummy_4, dummy_5) ;
The original input file was Sample.csv.
When I ran the LOAD cmd I got the error
Error 1261 Row 1 doesn't contain data for all columns.
Similar error for Rows 2 - 13.
I saved the CSV file as a Text file ( Sample.txt ) using Notepad. I got the same errors for the text file. It is only giving errors for the first 13 rows in the text file. So I deleted the first 13 rows and ran the LOAD again. I still get the same warnings for rows 1 to 13 only.
FYI, Each row does not have the same number of Fields.
Any help in resolving this would be appreciated.
The error message is pretty clear and just says that some of the lines in your CSV text file do not have 20 data points. You may try using the following regex pattern to highlight rows in the file which do not match:
^(?!(?:[^,]*,){19}[^,]*$).*$
Demo
Related
I have csv file with over a million lines and i need to write only the lines starting by '01' into database.
.csv file looks like this
01;104;5586;20;1000;
01;105;5586;80;1000;
01;106;5586;80;1000;
04;104;5586;20;1000;
06;105;5586;80;1000;
05;106;5586;80;1000;
SQL looks linke this
LOAD DATA LOCAL INFILE '$uploadfile'
REPLACE INTO TABLE mytable
FIELDS TERMINATED BY ';' ENCLOSED BY ''
IGNORE 1 LINES
(`a`, `b`, `c`, `d`, `e`);
So this works to import all lines. But how can i get only the lines which start with 01;....
You can try this one -
LOAD DATA LOCAL INFILE 'filename.csv'
REPLACE INTO TABLE mytable
FIELDS TERMINATED BY ';' ENCLOSED BY ''
LINES STARTING BY '01' TERMINATED BY '\r\n'
(`a`, `b`, `c`, `d`, `e`)
SET `a` = '01';
and you will get something like this -
01 104 5586 20 1000
01 105 5586 80 1000
01 106 5586 80 1000
Check line separator you use - '\r\n' or '\n' in TERMINATED BY clause.
Devart's solution would work if the data in the line never contains "01" again.
I found out that LINES STARTING BY is not working as expected according to this article:
https://bugs.mysql.com/bug.php?id=3632
"Another problem was that LINES STARTING BY xxx means that the MySQL will assume the lines start is at the next occurence of xxx".
So if my data also include 01, not just the start of the line, MySQL read some data from these lines. So this will insert bad data:
01;104;5586;01;1000
01;105;8586;80;1000
01;106;5586;80:0123
I wonder why there is no solution like the suggetions in the article from 2012:
(1) let LINES STARTING BY 'X' continue to mean "line contains 'X' anywhere and data prior to 'X' in the record is skipped" and document this as such.
(2) add LINES STARTING BY 'X' POSITION N which means "line contains 'X' beginning at character position N relative to the beginning of the record.
I ended up spliting the files or adding a prefix (X) at the start of each line.
The .csv like this
X01;104;5586;20;1000;
X01;105;5586;01;1000;
X01;106;5586;80;1000;
X04;104;5586;01;0123;
X06;105;5586;80;1000;
X05;106;5586;80;1000;
And the code like this:
LOAD DATA LOCAL INFILE 'filename.csv'
REPLACE INTO TABLE mytable
FIELDS TERMINATED BY ';' ENCLOSED BY ''
LINES STARTING BY 'X01' TERMINATED BY '\r\n'
(`a`, `b`, `c`, `d`, `e`) SET `a` = '01';
Maybe someone else out there had the same missunderstanding with LINES STARTING BY.
In phpMyAdmin, in the tab import, how to specify which columns of the csv must be skipped by the import?
For example, I have this csv:
col1 col2 col3 col4 col5
a b c x 0
1 2 3 y 1
I need to manually import that csv skipping the 4th column.
With the Csv using LOAD DATA I can specify the Column names; in that field I tried to insert these but without working:
col1, col2, col3, #dummy, col5
Invalid column (#dummy) specified! Ensure that columns names are spelled correctly, separated by commas, and not enclosed in quotes.
and
col1, col2, col3, , col5
SQL query:
LOAD DATA LOCAL INFILE '/tmp/phpBlaBlaBla'
INTO TABLE `tblName` FIELDS TERMINATED BY ';' ENCLOSED BY '"' ESCAPED BY '\' LINES TERMINATED BY '\n' IGNORE 1 LINES
(`col1` , `col2` , `col3` , , `col5`)
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near `col5`)' at line 1
The phpMyAdmin version is 4.0.10.16.
Thanks in advance!
I would take data from csv file into my table. I'm using command:
LOAD DATA INFILE 'C:\\...\\file.csv' INTO TABLE table FIELDS TERMINATED BY ',' IGNORE 1 LINES;
The problem is that I have data as below:
Col1,Col2,Col3,Col4
1,name11,name21,name31
2,"name21, aaa.",name22,name23
First row is ok, but second not, because "name21, aaa." is reading as two columns so I don't have name23 in table.
Any idea how can I resolve this problem?
You need to understand LOAD DATA INFILE syntax.
Check this link MySQL - LOAD DATA INFILE.
LOAD DATA LOCAL INFILE 'C:\\...\\file.csv' INTO TABLE table
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(col1, col2, col3, col4, col5...)
I have 35 CSV files which I want to import to MYSQL table(say 'test'). I want to create one column in 'test' table( say 'file_name'). This column will contain name of the CSV from which data has been imported. The file names are unique IDs, that is why I want to get file name as input in the table.
Suppose I have CSV files like X1.csv, X2.CSV, X3.csv .... X35.csv. I want a column in 'test' table as 'file_name' such that 'test' table looks something like:
col1 -> a, b, c, d
col2 -> x, y, w, z
...
...
... ....
file_name -> X1, X1, X2, X3
Note: I tried to search this question on forum but I could not find any suitable solution. Also I am new to MYSQL, please help even it is a trivial thing.
I'm not sure this is exactly what you are looking for, but at first sight, you should investigate the LOAD DATA INFILE statement:
LOAD DATA INFILE 'X1.csv' INTO TABLE tbl_name -- Load the content of the CSV file
FIELDS TERMINATED BY ',' ENCLOSED BY '"' -- assuming fields separate by ",", enclosed by "'"
LINES TERMINATED BY '\r\n' -- assuming end-of-line being '\r\n'
IGNORE 1 LINES -- assuming first line is a header and should be ignored
SET file_name = 'X1'; -- force the column `file_name` to be the name of the file
Please note that with such statement, each field will go in its own column of the table. And each line of the CSV data file will be loaded a one row in the table. This will imply that there will be several rows in the result table with the same file name. In fact one row per data line.
I've got a CSV file with 11 columns and I have a MySQL table with 9 columns.
The CSV file looks like:
col1, col2, col3, col4, col5, col6, col7, col8, col9, col10, col11
and the MySQL table looks like:
col1, col2, col3, col4, col5, col6, col7, col8, col9
I need to map the columns 1-8 of CSV file directly to the first 8 columns of the MySQL table. I then need to skip the next two columns in the CSV file and then map column 11 of CSV file to column 9 of MySQL table.
At the moment I am using the following SQL command:
LOAD DATA LOCAL INFILE 'filename.csv' INTO TABLE my_table
FIELDS TERMINATED BY ','
ENCLOSED BY ''
LINES TERMINATED BY '\n'
But the above code maps the first 9 columns of CSV file to the 9 columns in the MySQL table.
From Mysql docs:
You can also discard an input value by
assigning it to a user variable and
not assigning the variable to a table
column:
LOAD DATA INFILE 'file.txt'
INTO TABLE t1 (column1, #dummy, column2, #dummy, column3);
step1.deal with awk.
cat file.txt |awk '{print $1,$2,$5...}'>new_file.txt
step2.load into mysql.
load data local infile 'new_file' into table t1(...)
the method below is simple,but not allowed in lower version of mysql.
LOAD DATA INFILE 'file.txt'
INTO TABLE t1 (column1, #dummy, column2, #dummy, column3);
#deemi:
The only way to ignore the #dummy is by setting the field's Default to AUTO INCREMENT.
So you can skip the field and just code like this,
LOAD DATA INFILE 'file.txt'
INTO TABLE t1 (column2, column3, column4, column5);
//assumes that the fieldname column1 is set to AUTO INCREMENT by default.
I think there is one more change in the code:
The following SQL command:
LOAD DATA LOCAL INFILE 'filename.csv' INTO TABLE my_table
FIELDS TERMINATED BY ','
ENCLOSED BY ''
LINES TERMINATED BY '\n'
-will probably result in an data truncation error.
So it is better to use LINES TERMINATED BY '\r\n' instead of LINES TERMINATED BY '\n'
SO the code will be:
LOAD DATA LOCAL INFILE 'filename.csv' INTO TABLE my_table
FIELDS TERMINATED BY ','
ENCLOSED BY ''
LINES TERMINATED BY '\r\n'