In the other question, I asked about why non-empty values become NULL. But in this question, I asked about how I could make empty values become NULL but not zero.
I found that missing float values are always represented as 0, but not NULL. How could I change that?
The following are the codes with which I created the table and loaded data.
CREATE TABLE Products(sku INTEGER, name VARCHAR(255), description TEXT,
regularPrice FLOAT,
customerReviewAverage FLOAT default NULL );
LOAD DATA LOCAL INFILE 'product.csv'
INTO TABLE Products
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(sku, name, #description, regularPrice, customerReviewAverage)
SET description = IF(#description='',NULL,#description);
This is a sample of data in product.csv.
19658847,Glanzlichter - CD,,12.99,5.0
19658856,Glanzlichter - CD,,6.99,
19658865,Glanzlichter - CD,,8.99,
1965886,Beach Boys '69 - CASSETTE,,6.99,4.5
The issue is how MySQL interprets an empty field vs a missing field. From the docs for LOAD DATA INFILE...
If an input line has too few fields, the table columns for which input fields are missing are set to their default values.
An empty field value is interpreted different from a missing field. For string types, the column is set to the empty string. For numeric types, the column is set to 0.
In this case it seems MySQL considers it to be empty. You can see this from show warnings.
mysql> show warnings;
+---------+------+------------------------------------------------------------+
| Level | Code | Message |
+---------+------+------------------------------------------------------------+
| Warning | 1265 | Data truncated for column 'customerReviewAverage' at row 2 |
| Warning | 1265 | Data truncated for column 'customerReviewAverage' at row 3 |
+---------+------+------------------------------------------------------------+
2 rows in set (0.00 sec)
Whereas if we remove the trailing commas so the field is missing...
19658847,Glanzlichter - CD,,12.99,5.0
19658856,Glanzlichter - CD,,6.99
19658865,Glanzlichter - CD,,8.99
1965886,Beach Boys '69 - CASSETTE,,6.99,4.5
Then the data is set to null.
mysql> show warnings;
+---------+------+--------------------------------------------+
| Level | Code | Message |
+---------+------+--------------------------------------------+
| Warning | 1261 | Row 2 doesn't contain data for all columns |
| Warning | 1261 | Row 3 doesn't contain data for all columns |
+---------+------+--------------------------------------------+
mysql> select * from products;
+----------+---------------------------+-------------+--------------+-----------------------+
| sku | name | description | regularPrice | customerReviewAverage |
+----------+---------------------------+-------------+--------------+-----------------------+
| 19658847 | Glanzlichter - CD | NULL | 12.99 | 5 |
| 19658856 | Glanzlichter - CD | NULL | 6.99 | NULL |
| 19658865 | Glanzlichter - CD | NULL | 8.99 | NULL |
| 1965886 | Beach Boys '69 - CASSETTE | NULL | 6.99 | 4.5 |
+----------+---------------------------+-------------+--------------+-----------------------+
4 rows in set (0.00 sec)
Doing the same thing for #customerReviewAverage as for #description worked for me in MySQL 5.7.
LOAD DATA
LOCAL INFILE 'product.csv'
INTO TABLE Products
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(sku, name, #description, regularPrice, #customerReviewAverage)
SET description = IF(#description='',NULL,#description),
customerReviewAverage = IF(#customerReviewAverage='',NULL,#customerReviewAverage);
mysql> select * from products;
+----------+---------------------------+-------------+--------------+-----------------------+
| sku | name | description | regularPrice | customerReviewAverage |
+----------+---------------------------+-------------+--------------+-----------------------+
| 19658847 | Glanzlichter - CD | NULL | 12.99 | 5 |
| 19658856 | Glanzlichter - CD | NULL | 6.99 | NULL |
| 19658865 | Glanzlichter - CD | NULL | 8.99 | NULL |
| 1965886 | Beach Boys '69 - CASSETTE | NULL | 6.99 | 4.5 |
+----------+---------------------------+-------------+--------------+-----------------------+
4 rows in set (0.00 sec)
Related
When I'm trying to import csv into MySQL table, I'm getting an error
Data too long for column 'incident' at row 1
I'm sure the values are not higher than varchar(12). But, still I'm getting the error.
MariaDB [pagerduty]>
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv'
REPLACE INTO TABLE incidents
ignore 1 lines;
ERROR 1406 (22001): Data too long for column 'incident' at row 1
MariaDB [pagerduty]>
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv'
INTO TABLE incidents
ignore 1 lines;
ERROR 1406 (22001): Data too long for column 'incident' at row 1
While trying with REPLACE, the data is uploading only one column(which set on primary key)
MariaDB [pagerduty]>
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv'
IGNORE INTO TABLE incidents
ignore 1 lines;
Query OK, 246 rows affected, 1968 warnings (0.015 sec)
Records: 246 Deleted: 0 Skipped: 0 Warnings: 1968
**Columns:**
+----------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+-------+
| incident | varchar(12) | NO | PRI | NULL | |
| description | varchar(300) | YES | | NULL | |
| status | varchar(12) | YES | | NULL | |
| urgency | varchar(7) | YES | | NULL | |
| service | varchar(27) | YES | | NULL | |
| trigger | varchar(25) | YES | | NULL | |
| team | varchar(20) | YES | | NULL | |
| incident_start | datetime(6) | YES | | NULL | |
| incident_end | datetime(6) | YES | | NULL | |
| resolved_by | varchar(20) | YES | | NULL | |
+----------------+--------------+------+-----+---------+-------+
10 rows in set (0.003 sec)
By default, MySQL looks for a TAB character to separate values. Your file is using a comma, so MySQL reads the entire line and assumes it is the value for the first column only.
You need to tell MySQL that the column terminator is a comma, and while you're at it, tell it about the enclosing double quotes.
Try this:
LOAD DATA INFILE '/var/lib/mysql/pagerduty/script_output.csv' REPLACE INTO TABLE incidents
columns terminated by ','
optionally enclosed by '"'
ignore 1 lines;
Reference
If you THINK your data appears ok, and its still nagging about the too long data, how about creating a new temporary table structure and set your first column incident to a varchar( 100 ) just for grins... maybe even a few others if they too might be causing a problem.
Import the data to THAT table to see if same error or not.
If no error, then check the maximum trimmed length of the data in respective columns and analyze the data itself... bad format, longer than expected, etc.
Once resolved, then you can pull into production once you have figured it out. You could also always PRE-LOAD the data into this larger table structure, truncate it before each load so no dups via primary key on a fresh load.
I have had to do that in the past, also was efficient for pre-qualifying lookup table IDs for new incoming data. Additionally could apply data cleansing in the temp table before pulling into production.
I need convert in my table of DB MySQL the column values from VARCHAR(255) to DECIMAL (10,5).
I have tried this SQL query in which first replace ',' with '.' and then set new the column values to DECIMAL (10,5):
[SQL]UPDATE `doTable`
SET doAV = REPLACE (doAV, ',', '.')
WHERE
YEAR (`doDate`) IN (YEAR(CURDATE()));
Affected rows: 83169
Time: 1.846s
[SQL]
ALTER TABLE `doTable` MODIFY COLUMN doAV DECIMAL (10, 5);
[Err] 1292 - Truncated incorrect DECIMAL value: '2226,000'
But I have error:
[Err] 1292 - Truncated incorrect DECIMAL value: '2226,000'
How to do resolve this?
Can you help me?
Thank you in advance for any help, really appreciated.
#Edit 01
+----------+
| doAV |
+----------+
| 2226.000 |
| 2226.00 |
| 2225.750 |
| 2225.600 |
| 2225.533 |
| 2225.250 |
| 2225.000 |
| 2225.00 |
| 2224.800 |
| 2224.667 |
+----------+
#Edit 02
+-----------+
| doAV |
+-----------+
| 2226.000 |
| 792.000 |
| 338.400 |
| 13635.000 |
| 1438.125 |
| 220.800 |
| 172.000 |
| 253.000 |
| 4258.500 |
| 352.800 |
+-----------+
I think you updated only rows in YEAR (doDate) IN (YEAR(CURDATE())). Try to update all the rows first of all. Delete WHERE condition from your UPDATE and run it again.
Try to check only the whole part
SELECT *
FROM doTable
WHERE LENGTH(LEFT(doAV,IFNULL(NULLIF(LOCATE('.',doAV)-1,-1),LENGTH(doAV))))>5
DECIMAL(10, 5) => your whole part can containt only 5 digitals (10-5=5).
I have a table of the following format:
mysql> describe tweet_info;
+-----------+--------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+-------------------+-----------------------------+
| tweet_id | bigint(20) | NO | PRI | NULL | |
| user_id | bigint(20) | YES | | NULL | |
| tweet | varchar(140) | YES | | NULL | |
| timestamp | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| fav_count | int(11) | YES | | NULL | |
| lat | float | YES | | NULL | |
| longi | float | YES | | NULL | |
| hashtags | varchar(140) | YES | | NULL | |
+-----------+--------------+------+-----+-------------------+-----------------------------+
8 rows in set (0.00 sec)
and a file called mini.txt of the following schema:
<tweet_id> <user_id> <tweet_text> <timestamp> <favourite_count> <latitude> <longitude> <hashtags>
244435656850411520 522575984 #SGodoyAlmirall #hongostibetanos Sat Sep 08 14:02:56 +0000 2012 0 -70.29044372 -18.48140825 hongostibetanos
When I used the following query:
load data infile 'mini.txt'into table tweet_info fields terminated by '\t' lines terminated by '\n';
The query works fine and all lines in the file are inserted into my database. Just that the timestamp is not well handled and all of them stay null. Upon searching the internet a bit, I found that we can set the format of the timestamp as follows:
load data infile 'mini.txt' into table tweet_info fields terminated by '\t' lines terminated by '\n' (#var4) SET timestamp=STR_TO_DATE(#var4,'%a %b %d %H:%i:%s +0000 %Y');
However, this generates the following error:
ERROR 1062 (23000): Duplicate entry '0' for key 'PRIMARY'
This seems weird since
There are no duplicates in my file (I have manually checked on the small file on which I am presently running my command).
The first command didn't say anything about the duplication of entries and was OK.
I would be really grateful if someoen could help me out.
You need to list all the columns in the column list:
load data infile 'mini.txt'
into table tweet_info
fields terminated by '\t'
lines terminated by '\n'
(tweet_id, user_id, tweet_text, #var4, favourite_count, latitude, longitude, hashtags>)
SET timestamp=STR_TO_DATE(#var4,'%a %b %d %H:%i:%s +0000 %Y');
Your code was assigning the first column in the input file to #var4, convering that to a date, and then inserting a row with only the timestamp column specified. So it was defaulting all the other columns, and creating duplicate tweet_id = 0 rows.
mysql> select * from friends limit 10;
+----+------------------------+-------------------------+
| id | user | fri |
+----+------------------------+-------------------------+
|MHL_bKFgQ | NULL
| 2 | MWhR9LvOdRbqtu1I_DRFBg | 8Y2EN4XNNhnwssuPb31sJg
| 3 | MWhR9LvOdRbqtu1I_DRFBg | A1jPleJ99kXZ3t9wQ3np-g
| 4 | MWhR9LvOdRbqtu1I_DRFBg | resYiOoGkQg6q0qgtj_1GA
| 5 | MWhR9LvOdRbqtu1I_DRFBg | skl1OnkjMqD4GdFpVhU88Q
| 6 | MWhR9LvOdRbqtu1I_DRFBg | 3Ss2aqrSoO7WbxE2GcLlfQ
| 7 | MWhR9LvOdRbqtu1I_DRFBg | SGy3JDbhtzDTTBO7unqQxg
| 8 | MWhR9LvOdRbqtu1I_DRFBg | Aez4Y1F0m2ucIfmfzPZfjw
| 9 | MWhR9LvOdRbqtu1I_DRFBg | fT4jx_9cRWyKFROxv_MIrw
|10 | MWhR9LvOdRbqtu1I_DRFBg | QmZOAYM7ITbTdkTn6ay_ag
+----+------------------------+-------------------------+
10 rows in set (0.00 sec)
Command:
mysql> select * from friends where fri="3Ss2aqrSoO7WbxE2GcLlfQ";
Result:
Empty set (0.37 sec)
This is beyond my understanding. There is a valid data in the above table but when i try to retrieve it, it shows no rows.
Details of the table:
CREATE TABLE friends (
id INT NOT NULL,
user varchar(100) NOT NULL ,
fri varchar(100) ,
PRIMARY KEY (id)
)ENGINE=MyISAM;
I got the issue. I have loaded my file with following command:
LOAD DATA LOCAL INFILE 'file.txt' INTO TABLE friends fields terminated by ',' lines terminated by '\n';
But the last column has ending '\r\n'. So, \r was a garbage which was invisible. When i clean the file by removing \r using python, the SQL works as expected. Thank you guys for all your support.
I want to import data from excel to a mysql database using the command line client.
This is an example of how my csv-file is built:
Name 1 | 1 | 2 | 3 |
Name 2 | 1 | 2 | 3 |
Name 3 | 1 | 2 | 3 |
I'm using the code:
LOAD DATA LOCAL INFILE 'path to file.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
I get the "Query OK" and with this code the formatting on the table should be almost exactly as the csv-file but I get this result:
| NULL | NULL | NULL |
| NULL | NULL | NULL |
| NULL | NULL | NULL |
What is wrong?
seems u have used '|' in your csv file as delimiters instead of
comma, try the code as
`LOAD DATA LOCAL INFILE 'path to file.csv' INTO TABLE table_name FIELDS TERMINATED BY '|'LINES TERMINATED BY '\n';`
Assume the following table structure:
CREATE TABLE `table_name` (
`name` VARCHAR(20) CHARACTER SET utf8 DEFAULT NULL,
`value1` INT(11) DEFAULT NULL,
`value2` INT(11) DEFAULT NULL,
`value3` INT(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
and the file, csv-file.csv:
Name 1,1,2,3
Name 2,1,2,3
Name 3,1,2,3
when I run the statement:
mysql> LOAD DATA INFILE '/path/csv-file.csv'
-> INTO TABLE `table_name`
-> FIELDS TERMINATED BY ','
-> LINES TERMINATED BY '\n';
mysql> SELECT `name`, `value1`, `value2`, `value3`
FROM `table_name`;
get the following result:
+--------+--------+--------+--------+
| name | value1 | value2 | value3 |
+--------+--------+--------+--------+
| Name 1 | 1 | 2 | 3 |
| Name 2 | 1 | 2 | 3 |
| Name 3 | 1 | 2 | 3 |
+--------+--------+--------+--------+
3 rows in set (0.00 sec)