I have the following table :
desc tr_tmp;
+-------+------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+------+------+-----+---------+-------+
| aya | text | NO | | NULL | |
+-------+------+------+-----+---------+-------+
1 row in set (0.002 sec)
But When I load data from file via :
LOAD DATA INFILE '/home/mohsen/codes/haq/translation-tmp/ber.mensur.txt'
INTO TABLE tr_tmp
FIELDS TERMINATED BY ''
ENCLOSED BY '"'
LINES TERMINATED BY '\n' (aya);
I get the following warnings:
/usr/local/lib/python3.7/dist-packages/pymysql/cursors.py:329: Warning: (1265, "Data truncated for column 'aya' at row 1218")
self._do_get_result()
/usr/local/lib/python3.7/dist-packages/pymysql/cursors.py:329: Warning: (1265, "Data truncated for column 'aya' at row 1266")
self._do_get_result()
/usr/local/lib/python3.7/dist-packages/pymysql/cursors.py:329: Warning: (1265, "Data truncated for column 'aya' at row 1611")
After above warnings, It can't load rest of file.
Related
When I'm trying to import csv into MySQL table, I'm getting an error
Data too long for column 'incident' at row 1
I'm sure the values are not higher than varchar(12). But, still I'm getting the error.
MariaDB [pagerduty]>
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv'
REPLACE INTO TABLE incidents
ignore 1 lines;
ERROR 1406 (22001): Data too long for column 'incident' at row 1
MariaDB [pagerduty]>
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv'
INTO TABLE incidents
ignore 1 lines;
ERROR 1406 (22001): Data too long for column 'incident' at row 1
While trying with REPLACE, the data is uploading only one column(which set on primary key)
MariaDB [pagerduty]>
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv'
IGNORE INTO TABLE incidents
ignore 1 lines;
Query OK, 246 rows affected, 1968 warnings (0.015 sec)
Records: 246 Deleted: 0 Skipped: 0 Warnings: 1968
**Columns:**
+----------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+-------+
| incident | varchar(12) | NO | PRI | NULL | |
| description | varchar(300) | YES | | NULL | |
| status | varchar(12) | YES | | NULL | |
| urgency | varchar(7) | YES | | NULL | |
| service | varchar(27) | YES | | NULL | |
| trigger | varchar(25) | YES | | NULL | |
| team | varchar(20) | YES | | NULL | |
| incident_start | datetime(6) | YES | | NULL | |
| incident_end | datetime(6) | YES | | NULL | |
| resolved_by | varchar(20) | YES | | NULL | |
+----------------+--------------+------+-----+---------+-------+
10 rows in set (0.003 sec)
By default, MySQL looks for a TAB character to separate values. Your file is using a comma, so MySQL reads the entire line and assumes it is the value for the first column only.
You need to tell MySQL that the column terminator is a comma, and while you're at it, tell it about the enclosing double quotes.
Try this:
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv' REPLACE INTO TABLE incidents
columns terminated by ','
optionally enclosed by '"'
ignore 1 lines;
Reference
If you THINK your data appears ok, and its still nagging about the too long data, how about creating a new temporary table structure and set your first column incident to a varchar( 100 ) just for grins... maybe even a few others if they too might be causing a problem.
Import the data to THAT table to see if same error or not.
If no error, then check the maximum trimmed length of the data in respective columns and analyze the data itself... bad format, longer than expected, etc.
Once resolved, then you can pull into production once you have figured it out. You could also always PRE-LOAD the data into this larger table structure, truncate it before each load so no dups via primary key on a fresh load.
I have had to do that in the past, also was efficient for pre-qualifying lookup table IDs for new incoming data. Additionally could apply data cleansing in the temp table before pulling into production.
Recently updated to mysql Ver 8.0.19 for osx10.15 on x86_64 (Homebrew)
I am trying to load data from a tsv file to add entries to an existing table. Everything seems to be working fine, except the values which are "NULL" (as in it literally says that, it's not empty) are being converted to 0000-00-00 00:00:00. There are no errors but I do get the warning"Data truncated for column 'beginexp' at row 2".
Here's what I've tried, to no avail:
ALTER TABLE infomegatask
ALTER beginexp SET DEFAULT NULL; # same for field 'endhit'
ALTER TABLE infomegatask MODIFY COLUMN beginexp DATETIME NULL; #even though it was already nullable to begin with when I checked the schema
#also set sql_mode="NO_ZERO_DATE"
Here is a redacted snippet of what the tsv looks like (sensitive data has been obscured):
uniqueid assignmentid workerid hitid ipaddress browser platform language cond counterbalance codeversion beginhit beginexp endhit bonus status mode datastring
XXXXX:YYYYY ZZZZZ AAAAA BBBBB CCCCC chrome windows UNKNOWN 0 0 4.2 2020-04-22 16:32:24 NULL NULL 0 4 live
mysql> LOAD DATA LOCAL INFILE 'test/DB/tsv/infomegatask_round_9g.tsv' INTO TABLE infomegatask FIELDS TERMINATED BY '\t' ENCLOSED BY '' LINES TERMINATED BY '\n' IGNORE 1 LINES;
Query OK, 24 rows affected, 13 warnings (0.21 sec)
Records: 24 Deleted: 0 Skipped: 0 Warnings: 13
mysql> warnings
Show warnings enabled.
mysql> show warnings
-> ;
+---------+------+------------------------------------------------+
| Level | Code | Message |
+---------+------+------------------------------------------------+
| Warning | 1265 | Data truncated for column 'beginexp' at row 2 |
| Warning | 1265 | Data truncated for column 'endhit' at row 2 |
| Warning | 1265 | Data truncated for column 'beginexp' at row 4 |
| Warning | 1265 | Data truncated for column 'endhit' at row 4 |
| Warning | 1265 | Data truncated for column 'beginexp' at row 7 |
| Warning | 1265 | Data truncated for column 'endhit' at row 7 |
| Warning | 1265 | Data truncated for column 'beginexp' at row 9 |
| Warning | 1265 | Data truncated for column 'endhit' at row 9 |
| Warning | 1265 | Data truncated for column 'beginexp' at row 10 |
| Warning | 1265 | Data truncated for column 'endhit' at row 10 |
| Warning | 1265 | Data truncated for column 'endhit' at row 14 |
| Warning | 1265 | Data truncated for column 'beginexp' at row 20 |
| Warning | 1265 | Data truncated for column 'endhit' at row 20 |
+---------+------+------------------------------------------------+
13 rows in set (0.00 sec)
I mean the workaround I'm going with for now is to just update these zero datetimes to NULL, but it's a bit annoying and I would just like to figure out what is going wrong.
In the LOAD DATA syntax you can convert columns.
So in your case:
LOAD DATA
INTO TABLE xx ( .., #beginexp , ... )
...
SET beginexp = IF(#beginexp='NULL', NULL, #beginexp),
...
Mysql table price structure:
CREATE TABLE `price` (
`code` varchar(12) ,
`date` date ,
`open` decimal(8,2) ,
`high` decimal(8,2) ,
`low` decimal(8,2) ,
`close` decimal(8,2) ,
`amount` decimal(20,2) ,
`volume` decimal(16,2)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Here is the data.csv file which i want to load into tablbe price.
000046.XSHE,19940912,20.0,20.0,16.0,16.92,121262592.0,7043300
000046.XSHE,19940913,17.0,17.32,16.0,16.46,47195860.0,2810800
000046.XSHE,19940914,16.3,16.4,15.49,15.95,24762992.0,1558300
The max value for volume is 47195860.0 which is in the range of decimal(20,2).
Load it with mysql load command.
LOAD DATA local INFILE 'data.csv'
INTO TABLE finance.price
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
lines terminated by '\r\n';
An error info occurs:
Query OK, 1 row affected, 1 warning (0.09 sec)
Records: 1 Deleted: 0 Skipped: 0 Warnings: 1
Show it to get the reason.
show warnings;
+---------+------+---------------------------------------------+
| Level | Code | Message |
+---------+------+---------------------------------------------+
| Warning | 1265 | Data truncated for column 'volume' at row 1 |
+---------+------+---------------------------------------------+
1 row in set (0.00 sec)
Number 121262592.0 is so lower than decimal(20,2),why Data truncated for column 'volume' at row 1,only one row loaded.
select * from price;
+-------------+------------+-------+-------+-------+-------+--------------+------------+
| code | date | open | high | low | close | amount | volume |
+-------------+------------+-------+-------+-------+-------+--------------+------------+
| 000046.XSHE | 1994-09-12 | 20.00 | 20.00 | 16.00 | 16.92 | 121262592.00 | 7043300.00 |
+-------------+------------+-------+-------+-------+-------+--------------+------------+
1 row in set (0.00 sec)
LOAD DATA local INFILE 'data.csv'
INTO TABLE finance.price
FIELDS TERMINATED BY ','
OPTIONAL ENCLOSED BY '"'
lines terminated by '\r\n';
Create sample database and table to use with.
create database `test`;
use `test`;
create table `test` (`value` float(10,2) null);
insert into test (value) values (null);
select value from test;
+-------+
| value |
+-------+
| NULL |
+-------+
1 row in set (0.00 sec)
Now i want to create a txt file ,when you load it ,null data can be loaded into the test table.
delete from test;
Here is my data file named test.txt showed in vim(:set list).
Load the file into table.
LOAD DATA LOCAL INFILE "test.txt"
INTO TABLE test
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n' \w;
select value from test;
+-------+
| value |
+-------+
| 0.00 |
| 0.00 |
| 0.00 |
| 0.00 |
| 0.00 |
+-------+
How to load null data for value field with file in my case?
How to create this kind of data file,the value field is null after you load it?
The runtime environment:win10+mysql-5.7.
select version();
+-----------+
| version() |
+-----------+
| 5.7.22 |
+-----------+
1 row in set (0.03 sec)
Show test.txt file in vim with set list.
\N$
\N^I$
case:without set
Load it without (value) SET value = (CASE WHEN #value IS NULL THEN NULL ELSE #value END).
LOAD DATA LOCAL INFILE "f:/test.txt"
INTO TABLE test
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n' \W;
OUTPUT:
Warning (Code 1265): Data truncated for column 'value' at row 1
Warning (Code 1262): Row 2 was truncated; it contained more data than there were input columns
select * from test;
+-------+
| value |
+-------+
| 0.00 |
| NULL |
+-------+
case:with set
Load it with (value) SET value = (CASE WHEN #value IS NULL THEN NULL ELSE #value END).
delete from test;
LOAD DATA LOCAL INFILE "f:/test.txt"
INTO TABLE test
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
(value)SET value = (CASE WHEN #value IS NULL THEN NULL ELSE #value END) \W;
OUTPUT:
Warning (Code 1265): Data truncated for column 'value' at row 1
Warning (Code 1262): Row 2 was truncated; it contained more data than there were input columns
select * from test;
+-------+
| value |
+-------+
| NULL |
| NULL |
+-------+
Thank #Schwern.
LOAD DATA LOCAL INFILE "f:/test.txt"
INTO TABLE test
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\r\n' \W;
Either works fine with no warnings.
You can use \N.
The problem is all your input lines have an extra empty field. Tab terminates the field, like a comma. You've used it as if it encloses it like quotes. For example, your first line ^Inull^I is three fields. The first is empty. The second contains null. And the third is empty.
MySQL should have said something like
Query OK, 5 rows affected, 8 warnings (0.01 sec)
Records: 5 Deleted: 0 Skipped: 0 Warnings: 8
You can read those warnings with show warnings.
+---------+------+---------------------------------------------------------------------------+
| Level | Code | Message |
+---------+------+---------------------------------------------------------------------------+
| Warning | 1265 | Data truncated for column 'value' at row 1 |
| Warning | 1262 | Row 1 was truncated; it contained more data than there were input columns |
| Warning | 1265 | Data truncated for column 'value' at row 2 |
| Warning | 1262 | Row 2 was truncated; it contained more data than there were input columns |
| Warning | 1265 | Data truncated for column 'value' at row 3 |
| Warning | 1265 | Data truncated for column 'value' at row 4 |
| Warning | 1262 | Row 4 was truncated; it contained more data than there were input columns |
| Warning | 1265 | Data truncated for column 'value' at row 5 |
+---------+------+---------------------------------------------------------------------------+
All you need is this:
\N$
Note that lines terminated by is sensitive to Unix vs Windows newlines. From the docs...
If you have generated the text file on a Windows system, you might have to use LINES TERMINATED BY '\r\n' to read the file properly, because Windows programs typically use two characters as a line terminator.
This explains the warnings you are getting.
I made a table which name is 'test' in mysql in my PC like belows.
create table test(
telnum varchar(20) not null,
reg_date datetime not null default '0000-00-00 00:00:00',
remarks text,
primary key(telnum)
);
And I uploaded a file named 1.txt into table 'test'.
1.txt's contents are like belows :
01011112222
01022223333
01033334444
And 'load data infile' syntax are like belows :
load data infile "c:/temp/1.txt"
ignore into table test;
But there is a problem.
Every phone numbers were cut like below.
Query OK, 3 rows affected, 3 warnings (0.00 sec)
Records: 3 Deleted: 0 Skipped: 0 Warnings: 3
mysql> select * from test;
+-------------+---------------------+---------+
| telnum | reg_date | remarks |
+-------------+---------------------+---------+
|12222 | 0000-00-00 00:00:00 |
|23333 | 0000-00-00 00:00:00 |
|34444 | 0000-00-00 00:00:00 |
+-------------+---------------------+---------+
3 rows in set (0.00 sec)
Only 5 characters are remained from 11 characters.
6 characters disappeared.
And second problem is 'warnings'. Reason of warnings is like below.
mysql> show warnings;
+---------+------+---------------------------------------------------+
| Level | Code | Message |
+---------+------+---------------------------------------------------+
| Warning | 1264 | Out of range value for column 'reg_date' at row 1 |
| Warning | 1264 | Out of range value for column 'reg_date' at row 2 |
| Warning | 1264 | Out of range value for column 'reg_date' at row 3 |
+---------+------+---------------------------------------------------+
3 rows in set (0.00 sec)
I did'nt put anything into the reg_date field.
But the message says out of range value for column 'reg_date'.
What are the reasons and how can I solve these problems?
Try to change the line delimiter. On Windows it's usually \r\n rather then \n which is default if you omit LINES TERMINATED BY clause
LOAD DATA INFILE "/tmp/1.txt"
IGNORE INTO TABLE test
LINES TERMINATED BY '\n'