Problem: above mentioned load data query inserting 0 instead of null for price field(column)
Mysql Query :
LOAD DATA LOCAL INFILE '/tmp/data.csv'
REPLACE INTO TABLE bug_repeat
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';
file data.csv content is as below
name,\N
name,3
bug_repeat table structure:
CREATE TABLE `bug_repeat` (
`name` varchar(10) DEFAULT NULL,
`price` decimal(12,6) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Note: above query is not working on one of MYSQL server but exact same query working fine on two other MYSQL server. I don't know what going wrong. Can some please let me know what is the exact issue.( all MYSQL servers(version 5.7.22) are on Ubuntu 16.xxx OS). I am getting same problem for bigint data type as well.
show warnings result:
1265 (01000): Data truncated for column 'price' at row 1
Related
I have a .csv file which look like this:
and that can be downloaded from here.
I create my DB and my table with the following code:
CREATE DATABASE test_schema;
CREATE TABLE test_schema.teams (
teamkey SMALLINT NOT NULL AUTO_INCREMENT,
teamid CHAR(3) NOT NULL,
yearid YEAR(4) NOT NULL,
leagueid CHAR(2) NOT NULL,
teamrank TINYINT(2) NOT NULL,
PRIMARY KEY (teamkey),
UNIQUE KEY teamkey_UNIQUE (teamkey),
KEY teamid_yearid_leagueid_UNIQUE (teamid, yearid, leagueid),
CONSTRAINT check_teamrank CHECK (((teamrank >= 0) and (teamrank <= 12))),
CONSTRAINT check_year CHECK (((yearid >= 1871) and (yearid <=2155))))
ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
Now, when I try to import using:
LOAD DATA LOCAL INFILE "path_to_file_in_my_computer/Teams.csv"
INTO TABLE test_schema.teams
FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(#teamID, #yearID, #lgID, #Rank);
I get 2895 warnings which are all the same:
Warning (Code 3819): Check constraint 'check_year' is violated.
This warning makes no sense since yearid goes from 1871 to 2018 as can be corroborated if you look at the structure of the Teams.csv file. So any advice or suggestion on how to handle this error will be much appreciated. I'm working on MySQL Workbench 8.0.
PS: I posted a similar question (deleted) today morning but it needed more details that are provided here.
You don't have the column names in the correct order in the LOAD DATA query.
LOAD DATA LOCAL INFILE "path_to_file_in_my_computer/Teams.csv"
INTO TABLE test_schema.teams
FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(yearid, leagueid, teamid, #franchid, #divid, teamrank);
You can assign directly to the table column names, you don't need to use #yearID unless you have to do extra processing before storing in the table.
Hello! I am new to MYSQL so kindly explain in as simple language as possible!
I have a csv with 400k rows and want to import it into mysql. I am using LOAD DATA LOCAL INFILE command for this purpose:
LOAD DATA LOCAL INFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/Comorbidity Covid-19.csv'
INTO TABLE `comorbidity covid-19`
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;
The issue is that only about 200k records are being imported while the csv contains 400k records. Why is this happening? I executed the command both in the command prompt and in MySql Workbench but both give the same output. Also the date column is not being imported correctly. Instead of dates being displayed it is showing 0000-00-00 in each rows.
PS: OPT_LOCAL_INFILE=1 in manage database connections!
PS : Here is some sample data
What I did was first I created an empty table in the database with respective column types by. I created an empty table with only the column headers by right clicking on tables and selecting create new table option where I selected the proper type for each columns.. Date as of and Start Date were given Date type and so on. Then I executed the above query both in command prompt and workbench to import the rows.
show create table comorbidity gives this result:
CREATE TABLE `comorbidity` (
`Date as of` date NOT NULL,
`Start Date` date NOT NULL,
`State` varchar(20) NOT NULL,
`Condition group` varchar(50) NOT NULL,
`Condition` varchar(45) NOT NULL,
`Age group` varchar(15) NOT NULL,
`Covid19 deaths` int NOT NULL,
`Number of mentions` int NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
It may be that the date is not in the correct format that is why it looks wrong. Try to modify the field from excel to a correct format, or perform a DATE() function for the date.
On the subject of importing all the records, check if there is any character that interrupts the execution.
The reason only 200k records were being imported was because I was using:
LINES TERMINATED BY '\n'
When I changed it to:
LINES TERMINATED BY '\r\n'
All 400 k records were imported.
Running into Error Code: 1265 with the following code and data.
Data was exported as CSV.
The score column containing float data appears is source of issue as without this column the data loading works.
I have tried using Decimals and this works but I curious why floats do not work yet i'm working small figures.
CREATE TABLE dummy(
entry_id INT NOT NULL,
first_name VARCHAR(15) NOT NULL,
last_name VARCHAR(15) NOT NULL,
score FLOAT NOT NULL,
PRIMARY KEY (entry_id)
);
LOAD DATA INFILE "C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/test_data.csv"
INTO TABLE dummy
COLUMNS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES;
Below is the sample csv file:
entry_id,first_name,last_name,score
1,Joshua,Mbweka,3.56
2,Mary,Njoroge,25.23
3,Adam,Kimanzi,64.41
4,Eve,Faluna,56.13
5,Lester,Chuma,89.21
I am trying to load data for Q1 2012 from the below link
https://s3.amazonaws.com/capitalbikeshare-data/index.html
My code is as follows:-
DROP DATABASE IF EXISTS bike;
CREATE DATABASE bike;
USE bike;
DROP TABLE IF EXISTS bike_2012;
CREATE TABLE bike_2012(
bike_duration INT NULL,
bike_start_date TIMESTAMP NULL,
bike_end_date TIMESTAMP NULL,
bike_s_station_no INT(5) NULL,
bike_s_station_name VARCHAR(255) NULL,
bike_e_station_no INT(5) NULL,
bike_e_station_name VARCHAR(255) NULL,
bike_number CHAR(6) NULL,
bike_member_type VARCHAR(25) NULL,
bike_ride_number INT auto_increment PRIMARY KEY);
LOAD DATA LOCAL INFILE 'C:/LAGASA_2018/MSBA/Data_Sources/2012-capitalbikeshare-tripdata/2012Q1-capitalbikeshare-tripdata.csv'
INTO TABLE bike_2012
FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '/n'
('bike_duration', #bike_start_date, #bike_end_date, 'bike_s_station_no','bike_s_station_name',
'bike_e_station_no','bike_e_station_name','bike_number','bike_member_type')
SET 'bike_start_date' = STR_TO_DATE(#bike_start_date, '%c/%e/%Y')
SET 'bike_end_date' = STR_TO_DATE(#bike_end_date, '%c/%e/%Y')
IGNORE 1 LINES;
SELECT * FROM bike_2012 LIMIT 10;
I am facing the following issues:-
Some columns that have integer data also have string data, so those parts are not getting loaded correctly. I tried to add OPTIONALLY ENCLOSED BY '"' but its not working.
Unable to change date to SQL date format
Other errors like Row doesn't contain data for all columns and data truncated for date columns are appearing.
I have been struggling to correct this. Please help.
Thanks and Regards
You won't be able to simply load wrong CSV into DB and fix it.
If you have access to PHP/Python or other language that has a driver to connect to your db engines, load that file into an array, or use something similar to fgets() in php to load it line by line and process each row separately, fix/convert data and then push it to db engine (I would suggest even grouping inserts for speed).
You are dealing not only with conversion, but there might be issues with string encoding (you didn't specify any in your CREATE TABLE which might cause a problem in itself.
There is a CSV file with the following data:
1;8-25-2010;0:05;210;4
2;8-25-2010;2:45;412;5
3;8-25-2010;3:40;300;3
4;8-25-2010;4:45;226;6
5;8-25-2010;5:20;206;4
6;8-25-2010;5:25;216;3
And there is MySQL Table:
CREATE TABLE IF NOT EXISTS `Schedule` (
`ID` SMALLINT NOT NULL AUTO_INCREMENT,
`Num` INT(10),
`PlannedDate` DATE,
`PlannedTime` TIME NOT NULL,
`resQty` INT(3) NOT NULL,
`stID` VARCHAR(10) NOT NULL,
PRIMARY KEY (`ID`),
FOREIGN KEY `stID` (`stID`) REFERENCES Stands (`stID`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Now I need to fill this table with the data from CSV file. For this I'm using the following code:
TRUNCATE TABLE testDB.Schedule;
LOAD DATA LOCAL INFILE 'C:\\temp\\Input.csv'
INTO TABLE testDB.Schedule FIELDS TERMINATED BY ';' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n' (Num,PlannedDate,PlannedTime,stID,resQty);
But the error message says that "Data truncated for column PlannedDate at row1", ErrorNr. 1265. The same error message for all rows.
The date has the wrong format, it should be 2012-01-19
If you are stuck with that date format (which is not the MySQL default), you can load those dates into a temporary variable and then convert it to a date, like this:
TRUNCATE TABLE testDB.Schedule;
LOAD DATA LOCAL INFILE 'C:\\temp\\Input.csv'
INTO TABLE testDB.Schedule FIELDS TERMINATED BY ';' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
(Num,#PlannedDate,PlannedTime,stID,resQty)
SET PlannedDate = STR_TO_DATE(#PlannedDate,'%m-%d-%Y');
in this line:
(Num,PlannedDate,PlannedTime,stID,resQty);
the two last parameters are reversed: stID,resQty
it SHOULD be:
resQty, stID
This is why you are getting a truncation error.