I'm trying to get this CSV file that I exported from excel loaded into my database and I can't seem to get the formatting correct no matter what I try.
Here is the SQL:
LOAD DATA INFILE 'path/file.csv'
INTO TABLE tbl_name
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(column1, column2, column3);
This works fine but then I run into trouble when the end of a line (column 3) ends in a quote. For example:
Actual value: These are "quotes"
Value in CSV: "These are ""quotes"""
What happens is that I will get an extra quote on that value in the database and also any additional lines until it reaches another quote in the CSV. Any ideas on how to solve this?
Hmm. I tried to duplicate this problem but can't. Where does my data differ from yours? Can you provide sample data to duplicate this? Here's what I did:
> cat /tmp/data.csv
"aaaa","bbb ""ccc"" ddd",xxx
xxx,yyy,"zzz ""ooo"""
foo,bar,baz
mysql> CREATE TABLE t2 (a varchar(20), b varchar(20), c varchar(20));
Query OK, 0 rows affected (0.01 sec)
mysql> LOAD DATA INFILE '/tmp/data.csv' INTO TABLE t2 FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' (a, b, c);
Query OK, 3 rows affected (0.00 sec)
Records: 3 Deleted: 0 Skipped: 0 Warnings: 0
mysql> select * from t2;
+------+---------------+-----------+
| a | b | c |
+------+---------------+-----------+
| aaaa | bbb "ccc" ddd | xxx |
| xxx | yyy | zzz "ooo" |
| foo | bar | baz |
+------+---------------+-----------+
3 rows in set (0.00 sec)
Looks ok to me(?)
Also note that if you're working on a Windows platform you might need to use
LINES TERMINATED BY '\r\n' instead.
Related
I have an hashes table with 2 columns, hash | plain
And a text file looking like that:
acbd18db4cc2f85cedef654fccc4a4d8:foo
37b51d194a7513e45b56f6524f2d51f2:bar
4e99e8c12de7e01535248d2bac85e732:foo:bar
I'm trying to execute this query:
LOAD DATA LOCAL INFILE 'file.txt' INTO TABLE hashes COLUMNS TERMINATED BY ':' LINES TERMINATED BY '\n'
The issue is, for the hash 4e99e8c12de7e01535248d2bac85e732, it will only insert foo, not foo:bar, because of COLUMNS TERMINATED BY ':'.
How can I make it "only split once" to fix this issue?
You could load into a user variable and use a bit of string maniplulation.
drop table if exists t;
create table t
(hash varchar(100),plain varchar(100));
LOAD DATA INFILE 'file.txt'
INTO TABLE t
LINES TERMINATED BY '\r\n'
(#var)
set
hash = substring_index(#var,':',1),
plain = replace(#var,substring_index(#var,':',1),'')
;
select *
from t;
+--------+----------+
| hash | plain |
+--------+----------+
| abc | :def |
| abc | :ghi |
| abc | :def:ghi |
+--------+----------+
3 rows in set (0.001 sec)
Note I have used \r\n to load this properly - you should test for your environment
I have a 350MB file named text_file.txt containing this tab delimited data:
345868230 1646198120 1531283146 Keyword_1531283146 1.55 252910000
745345566 1646198120 1539847239 another_1531276364 2.75 987831000
...
MySQL Database name: Xml_Date
Database table: PerformanceReport
I have already created the table with all the destination fields.
I want to import this text file data into a MySQL. I googled and found some commands like LOAD DATA INFILE and quite confused on how to use it.
How can I import this text file data?
It should be as simple as...
LOAD DATA INFILE '/tmp/mydata.txt' INTO TABLE PerformanceReport;
By default LOAD DATA INFILE uses tab delimited, one row per line, so should take it in just fine.
Walkthrough on using MySQL's LOAD DATA command:
Create your table:
CREATE TABLE foo(myid INT, mymessage VARCHAR(255), mydecimal DECIMAL(8,4));
Create your tab delimited file (note there are tabs between the columns):
1 Heart disease kills 1.2
2 one out of every two 2.3
3 people in America. 4.5
Use the load data command:
LOAD DATA LOCAL INFILE '/tmp/foo.txt'
INTO TABLE foo COLUMNS TERMINATED BY '\t';
If you get a warning that this command can't be run, then you have to enable the --local-infile=1 parameter described here: How can I correct MySQL Load Error
The rows get inserted:
Query OK, 3 rows affected (0.00 sec)
Records: 3 Deleted: 0 Skipped: 0 Warnings: 0
Check if it worked:
mysql> select * from foo;
+------+----------------------+-----------+
| myid | mymessage | mydecimal |
+------+----------------------+-----------+
| 1 | Heart disease kills | 1.2000 |
| 2 | one out of every two | 2.3000 |
| 3 | people in America. | 4.5000 |
+------+----------------------+-----------+
3 rows in set (0.00 sec)
How to specify which columns to load your text file columns into:
Like this:
LOAD DATA LOCAL INFILE '/tmp/foo.txt' INTO TABLE foo
FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'
(#col1,#col2,#col3) set myid=#col1,mydecimal=#col3;
The file contents get put into variables #col1, #col2, #col3. myid gets column 1, and mydecimal gets column 3. If this were run, it would omit the second row:
mysql> select * from foo;
+------+-----------+-----------+
| myid | mymessage | mydecimal |
+------+-----------+-----------+
| 1 | NULL | 1.2000 |
| 2 | NULL | 2.3000 |
| 3 | NULL | 4.5000 |
+------+-----------+-----------+
3 rows in set (0.00 sec)
If your table is separated by others than tabs, you should specify it like...
LOAD DATA LOCAL
INFILE '/tmp/mydata.txt' INTO TABLE PerformanceReport
COLUMNS TERMINATED BY '\t' ## This should be your delimiter
OPTIONALLY ENCLOSED BY '"'; ## ...and if text is enclosed, specify here
The LOAD DATA INFILE statement reads rows from a text file into a table at a very high speed.
LOAD DATA INFILE '/tmp/test.txt'
INTO TABLE test
FIELDS TERMINATED BY ','
LINES STARTING BY 'xxx';
If the data file looks like this:
xxx"abc",1
something xxx"def",2
"ghi",3
The resulting rows will be ("abc",1) and ("def",2). The third row in the file is skipped because it does not contain the prefix.
LOAD DATA INFILE 'data.txt'
INTO TABLE tbl_name
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
You can also load data files by using the mysqlimport utility; it operates by sending a LOAD DATA INFILE statement to the server
mysqlimport -u root -ptmppassword --local test employee.txt
test.employee: Records: 3 Deleted: 0 Skipped: 0 Warnings: 0
You should set the option:
local-infile=1
into your [mysql] entry of my.cnf file or call mysql client with the --local-infile option:
mysql --local-infile -uroot -pyourpwd yourdbname
You have to be sure that the same parameter is defined into your [mysqld] section too to enable the "local infile" feature server side.
It's a security restriction.
LOAD DATA LOCAL INFILE '/softwares/data/data.csv' INTO TABLE tableName;
LOAD DATA INFILE '/home/userlap/data2/worldcitiespop.txt' INTO TABLE cc FIELDS TERMINATED BY ','LINES TERMINATED BY '\r \n' IGNORE 1 LINES;
IGNORE 1 LINES to skip over an initial header line containing column names
FIELDS TERMINATED BY ',' is to read the comma-delimited file
If you have generated the text file on a Windows system, you might have to use LINES TERMINATED BY '\r\n' to read the file properly, because Windows programs typically use two characters as a line terminator. Some programs, such as WordPad, might use \r as a line terminator when writing files. To read such files, use LINES TERMINATED BY '\r'.
For me just adding the "LOCAL" Keyword did the trick, please see the attached image for easier solution.
My attached image contains both use cases:
(a) Where I was getting this error.
(b) Where error was resolved by just adding "Local" keyword.
Make Sure your Local-Infile variable is set to True (ON)
mysql> show global variables like 'local_infile';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| local_infile | OFF |
+---------------+-------+
1 row in set (0.04 sec)
mysql> set global local_infile=true;
Query OK, 0 rows affected (0.01 sec)
Find the correct path to store the txt files for loading in SQL tables
mysql> SELECT ##GLOBAL.secure_file_priv;
+------------------------------------------------+
| ##GLOBAL.secure_file_priv |
+------------------------------------------------+
| C:\ProgramData\MySQL\MySQL Server 8.0\Uploads\ |
+------------------------------------------------+
1 row in set (0.00 sec)
Load using data infile from the path (Use backward slashes in path)
mysql> load data infile 'C:/ProgramData/MySQL/MySQL Server
8.0/Uploads/text_file.txt' into table TABLE_NAME fields terminated by '\t' lines terminated by '\n';
1. if it's tab delimited txt file:
LOAD DATA LOCAL INFILE 'D:/MySQL/event.txt' INTO TABLE event
LINES TERMINATED BY '\r\n';
2. otherwise:
LOAD DATA LOCAL INFILE 'D:/MySQL/event.txt' INTO TABLE event
FIELDS TERMINATED BY 'x' (here x could be comma ',', tab '\t', semicolon ';', space ' ')
LINES TERMINATED BY '\r\n';
It is my test-tab.csv as below.
\data\ hello
Notice:there is a \t between \ and h in test-tab.csv，that is to say ,showing in vim(set list).
\data\^Ihello$
Prepare for loading data.
create table tab(`f1` varchar(10),`f2` varchar(10));
Load the data into table tab.
LOAD DATA LOCAL INFILE "f:/test-tab.csv"
INTO TABLE tab
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\r\n' \W;
And have a look.
select * from tab;
+------------+------+
| f1 | f2 |
+------------+------+
| data hello| NULL |
+------------+------+
1 row in set (0.000 sec)
How can i load data into table tab as below.
select * from tab;
+------------+------+
| f1 | f2 |
+------------+------+
| \data\ | hello|
+------------+------+
1 row in set (0.000 sec)
Constraint condition：keep the data format in test-tab.csv unchanged.
The problem seems to be escaping the tab.
FIELDS TERMINATED BY '\t' ESCAPED BY ''
If that does not work, is there some character that is not used in the data? Say |? Then
FIELDS TERMINATED BY '\t' ESCAPED BY '|'
LOAD DATA LOCAL INFILE "f:/test-tab.csv"
INTO TABLE tab
FIELDS TERMINATED BY '\t' ESCAPED BY ''
LINES TERMINATED BY '\n' \W;
1.add ESCAPED BY ''
2.LINES TERMINATED BY '\n'
The csv file was created in linux,i load it in my win-os.
I have a 350MB file named text_file.txt containing this tab delimited data:
345868230 1646198120 1531283146 Keyword_1531283146 1.55 252910000
745345566 1646198120 1539847239 another_1531276364 2.75 987831000
...
MySQL Database name: Xml_Date
Database table: PerformanceReport
I have already created the table with all the destination fields.
I want to import this text file data into a MySQL. I googled and found some commands like LOAD DATA INFILE and quite confused on how to use it.
How can I import this text file data?
It should be as simple as...
LOAD DATA INFILE '/tmp/mydata.txt' INTO TABLE PerformanceReport;
By default LOAD DATA INFILE uses tab delimited, one row per line, so should take it in just fine.
Walkthrough on using MySQL's LOAD DATA command:
Create your table:
CREATE TABLE foo(myid INT, mymessage VARCHAR(255), mydecimal DECIMAL(8,4));
Create your tab delimited file (note there are tabs between the columns):
1 Heart disease kills 1.2
2 one out of every two 2.3
3 people in America. 4.5
Use the load data command:
LOAD DATA LOCAL INFILE '/tmp/foo.txt'
INTO TABLE foo COLUMNS TERMINATED BY '\t';
If you get a warning that this command can't be run, then you have to enable the --local-infile=1 parameter described here: How can I correct MySQL Load Error
The rows get inserted:
Query OK, 3 rows affected (0.00 sec)
Records: 3 Deleted: 0 Skipped: 0 Warnings: 0
Check if it worked:
mysql> select * from foo;
+------+----------------------+-----------+
| myid | mymessage | mydecimal |
+------+----------------------+-----------+
| 1 | Heart disease kills | 1.2000 |
| 2 | one out of every two | 2.3000 |
| 3 | people in America. | 4.5000 |
+------+----------------------+-----------+
3 rows in set (0.00 sec)
How to specify which columns to load your text file columns into:
Like this:
LOAD DATA LOCAL INFILE '/tmp/foo.txt' INTO TABLE foo
FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'
(#col1,#col2,#col3) set myid=#col1,mydecimal=#col3;
The file contents get put into variables #col1, #col2, #col3. myid gets column 1, and mydecimal gets column 3. If this were run, it would omit the second row:
mysql> select * from foo;
+------+-----------+-----------+
| myid | mymessage | mydecimal |
+------+-----------+-----------+
| 1 | NULL | 1.2000 |
| 2 | NULL | 2.3000 |
| 3 | NULL | 4.5000 |
+------+-----------+-----------+
3 rows in set (0.00 sec)
If your table is separated by others than tabs, you should specify it like...
LOAD DATA LOCAL
INFILE '/tmp/mydata.txt' INTO TABLE PerformanceReport
COLUMNS TERMINATED BY '\t' ## This should be your delimiter
OPTIONALLY ENCLOSED BY '"'; ## ...and if text is enclosed, specify here
The LOAD DATA INFILE statement reads rows from a text file into a table at a very high speed.
LOAD DATA INFILE '/tmp/test.txt'
INTO TABLE test
FIELDS TERMINATED BY ','
LINES STARTING BY 'xxx';
If the data file looks like this:
xxx"abc",1
something xxx"def",2
"ghi",3
The resulting rows will be ("abc",1) and ("def",2). The third row in the file is skipped because it does not contain the prefix.
LOAD DATA INFILE 'data.txt'
INTO TABLE tbl_name
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
You can also load data files by using the mysqlimport utility; it operates by sending a LOAD DATA INFILE statement to the server
mysqlimport -u root -ptmppassword --local test employee.txt
test.employee: Records: 3 Deleted: 0 Skipped: 0 Warnings: 0
You should set the option:
local-infile=1
into your [mysql] entry of my.cnf file or call mysql client with the --local-infile option:
mysql --local-infile -uroot -pyourpwd yourdbname
You have to be sure that the same parameter is defined into your [mysqld] section too to enable the "local infile" feature server side.
It's a security restriction.
LOAD DATA LOCAL INFILE '/softwares/data/data.csv' INTO TABLE tableName;
LOAD DATA INFILE '/home/userlap/data2/worldcitiespop.txt' INTO TABLE cc FIELDS TERMINATED BY ','LINES TERMINATED BY '\r \n' IGNORE 1 LINES;
IGNORE 1 LINES to skip over an initial header line containing column names
FIELDS TERMINATED BY ',' is to read the comma-delimited file
If you have generated the text file on a Windows system, you might have to use LINES TERMINATED BY '\r\n' to read the file properly, because Windows programs typically use two characters as a line terminator. Some programs, such as WordPad, might use \r as a line terminator when writing files. To read such files, use LINES TERMINATED BY '\r'.
For me just adding the "LOCAL" Keyword did the trick, please see the attached image for easier solution.
My attached image contains both use cases:
(a) Where I was getting this error.
(b) Where error was resolved by just adding "Local" keyword.
Make Sure your Local-Infile variable is set to True (ON)
mysql> show global variables like 'local_infile';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| local_infile | OFF |
+---------------+-------+
1 row in set (0.04 sec)
mysql> set global local_infile=true;
Query OK, 0 rows affected (0.01 sec)
Find the correct path to store the txt files for loading in SQL tables
mysql> SELECT ##GLOBAL.secure_file_priv;
+------------------------------------------------+
| ##GLOBAL.secure_file_priv |
+------------------------------------------------+
| C:\ProgramData\MySQL\MySQL Server 8.0\Uploads\ |
+------------------------------------------------+
1 row in set (0.00 sec)
Load using data infile from the path (Use backward slashes in path)
mysql> load data infile 'C:/ProgramData/MySQL/MySQL Server
8.0/Uploads/text_file.txt' into table TABLE_NAME fields terminated by '\t' lines terminated by '\n';
1. if it's tab delimited txt file:
LOAD DATA LOCAL INFILE 'D:/MySQL/event.txt' INTO TABLE event
LINES TERMINATED BY '\r\n';
2. otherwise:
LOAD DATA LOCAL INFILE 'D:/MySQL/event.txt' INTO TABLE event
FIELDS TERMINATED BY 'x' (here x could be comma ',', tab '\t', semicolon ';', space ' ')
LINES TERMINATED BY '\r\n';
In MySQL, when I try to insert a backslash into my table, it does not accept it and gives me the content without the backslash.
id is set to auto increment:
Code:
INSERT INTO gender (sex, date) VALUES (
'male are allowed \ female are not allowed', "2012-10-06")
How do I insert a literal backslash?
Notes about escape sequences:
Escape Sequence Character Represented by Sequence
\0 An ASCII NUL (0x00) character.
\' A single quote (“'”) character.
\" A double quote (“"”) character.
\b A backspace character.
\n A newline (linefeed) character.
\r A carriage return character.
\t A tab character.
\Z ASCII 26 (Control+Z). See note following the table.
\\ A backslash (“\”) character.
\% A “%” character. See note following the table.
\_ A “_” character. See note following the table.
You need to escape your backslash :
INSERT INTO gender
(sex, date) VALUES (
'male are allowed \\ female are not allowed',
"2012-10-06")
Reference (with the list of all characters you must escape for mysql)
Handle backslashes and all control characters in mysql load data infile tool:
Step 1, create your table:
mysql> create table penguin (id int primary key, chucknorris VARCHAR(4000));
Query OK, 0 rows affected (0.01 sec)
Step 2, create your file to import and put this data in there.
1 spacealiens are on route
2 scramble the nimitz\
3 \its species 8472
4 \\\\\\\\\\\\\\\\\\
5 Bonus characters:!##$%^&*()_+=-[]\|}{;'":/.?>,< anything but tab
Step 3, insert into your table:
mysql> load data local infile '/home/el/foo/textfile.txt' into table penguin
fields terminated by '\t' lines terminated by '\n'
(#col1, #col2) set id=#col1, chucknorris=#col2;
Query OK, 4 rows affected, 1 warning (0.00 sec)
Records: 4 Deleted: 0 Skipped: 0 Warnings: 1
Step 4, and of course, it causes this strange output:
mysql> select * from penguin;
+----+-----------------------------------------------------------------+
| id | chucknorris |
+----+-----------------------------------------------------------------+
| 1 | spacealiens are on route |
| 2 | scramble the nimitz |
| 3 | |
| 4 | \\\\\\\\\ |
| 5 | Bonus characters:!##$%^&*()_+=-[]|}{;'":/.?>,< anything but tab |
+----+-----------------------------------------------------------------+
Step 5, analyze the warning:
mysql> show warnings;
+---------+------+--------------------------------------------------------+
| Level | Code | Message |
+---------+------+------------------------------------- ------------------+
| Warning | 1262 | Row 2 was truncated; it contained more data than there |
| | | were input columns |
+---------+------+--------------------------------------------------------+
1 row in set (0.00 sec)
Step 6, think about exactly what went wrong:
The backslash to the left of nimitz caused the mysql load data parser to concatenate the end of line 2 with the beginning of line 3. Then it bumped up against a tab and put 'scramble the nimitz\n3 into row 2.
The rest of row 3 is skipped because the extra words its species 8472 do not fit anywhere, it produces the warning you see above.
Row 4 had 18 backslashes, so there is no problem, and shows up as 9 backslahes because each was escaped. Had there been an odd number, the error on row 2 would have happened to row 4.
The bonus characters on row 5 came through normally. Everything is allowed except tab.
Step 7, reset table penguin:
mysql> delete from penguin;
Step 8, load into your table with the fields escaped by clause:
mysql> load data local infile '/home/el/foo/textfile.txt' into table penguin
fields terminated by '\t' escaped by '\b'
lines terminated by '\n' (#col1, #col2) set id=#col1,
chucknorris=#col2;
Query OK, 5 rows affected (0.00 sec)
Records: 5 Deleted: 0 Skipped: 0 Warnings: 0
Step 9, select from your table, interpret the results:
mysql> select * from penguin;
+----+------------------------------------------------------------------+
| id | chucknorris |
+----+------------------------------------------------------------------+
| 1 | spacealiens are on route |
| 2 | scramble the nimitz\ |
| 3 | \its species 8472 |
| 4 | \\\\\\\\\\\\\\\\\\ |
| 5 | Bonus characters:!##$%^&*()_+=-[]\|}{;'":/.?>,< anything but tab |
+----+------------------------------------------------------------------+
5 rows in set (0.00 sec)
And now everything is as we expect. The backslash at the end of line 2 does not escape the newline. The backslash before i on row 3 doesn't do anything. The 18 backslashes on row 4 are not escaped. And the bonus characters come through ok.
you can use this code :
$yourVariable = addcslashes($_POST["your param"],"\\");
for example in my web form i want insert local directory :
$localAddress = addcslashes($_POST["localAddress"],"\\");