In my dataset, there are many nonsense values of "#N/A". I create a new table and load the local CSV file into my new table. But because of the nonsense value, it shows the error as shown in the headline. How can I load a dataset successfully without the nonsense values?
Here is my code:
CREATE TABLE movies (
Yearnum INT NOT NULL,
Length INT,
Title VARCHAR(255) NOT NULL,
Subjct VARCHAR(255) NOT NULL,
Actor VARCHAR(255) NOT NULL,
Actress VARCHAR(255) ,
Director VARCHAR(255) ,
Popularity VARCHAR(255),
Awards VARCHAR(255) NOT NULL
);
LOAD DATA INFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/movies - movies.csv'
INTO TABLE movies
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;
Here is my link url of dataset:https://docs.google.com/spreadsheets/d/1J17LYPJZaW5QWQuQVJQonpJXGUbPGOJjg5bxj3SMupQ/edit?usp=sharing
You can work around this by using a column list with variables in the place of columns whose data might not be valid in the CSV file; the column values can then be made valid using a SET clause. For your example:
LOAD DATA INFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/movies - movies.csv'
INTO TABLE movies
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(Yearnum, #length, Title, Subjct, Actor, Actress, Director, Popularity, Awards)
SET Length = CASE WHEN #length = '#N/A' THEN 0 ELSE #length END
See the manual for more details.
Related
I'am trying to import data from csv i have a date data type which in the csv file is saved as %d-%b-%y (ex. 12-Aug-20)
The table
create table shows(
ShowID int unique,
Title varchar(255),
TypeID int,
Director varchar(255),
Cast blob,
DateAdded date,
year year,
Violence varchar(255),
Duration varchar(255),
Description blob
);
I tried running this script to populate:
INTO TABLE shows
FIELDS TERMINATED BY ','
optionally enclosed by '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS
(#ShowID,#type1,shows.Title,shows.Director,shows.`Cast`,#country,#date_added,shows.`year`,shows.Violence,shows.Duration,#c,shows.Description)
SET shows.ShowID =#ShowID ,
shows.TypeID = (select TypeID from typecountry t where t.`Type`=#type1 and t.Country=#country),
shows.DateAdded= STR_TO_DATE(#date_added , '%e-%b-%y');
and this error shows up :
SQL Error [1411] [HY000]: Incorrect datetime value: '' for function str_to_date
You have value that are empty i your table, sp you need to repalce them
LOAD DATA INFILE 'c:/tmp/myfile.csv'
INTO TABLE shows
FIELDS TERMINATED BY ','
optionally enclosed by '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS
(#ShowID,#type1,shows.Title,shows.Director,shows.`Cast`,#country,#date_added,shows.`year`,shows.Violence,shows.Duration,#c,shows.Description)
SET shows.ShowID =#ShowID ,
shows.TypeID = (select TypeID from typecountry t where t.`Type`=#type1 and t.Country=#country),
shows.DateAdded= STR_TO_DATE(IF (#date_added = '','01-01-1999', #date_added) , '%e-%b-%y');
I'm importing a CSV file into Heidi SQL. I've created the table using this code:
create table all_data
(
Keyword varchar(1000),
Position int,
Previous_Position int,
Search_Volume int,
KW_Difficulty float,
URL varchar(1000),
Post_title varchar(1000),
Post_URL varchar(1000),
Genre varchar(1000),
Location varchar(1000),
Avg_Daily_Visitors float,
pageviews int
)
;
but in the Avg_Daily_visitors column it has "\N" where there is no value. I've been importing the data with this code:
load data local infile 'C:/filepath.../All_Data.csv'
replace into table all_data
fields terminated by ','
enclosed by '"'
escaped by '"'
lines terminated by "\r\n"
ignore 1 rows
set
Avg_Daily_Visitors = replace(Avg_Daily_Visitors,"\N",0),
pageviews = replace(pageviews,"\N", 0)
;
but it's not replacing the values with 0, which is what I want to achieve. How do I make Heidi SQL replace "\N" with "0" on import?
Thanks.
First assign the value you read to a variable, then work on that variable. For this you specify the columns of your destination table, but a variable instead of the column where you want to replace.
load data local infile 'C:/filepath.../All_Data.csv'
replace into table all_data
fields terminated by ','
enclosed by '"'
escaped by '"'
lines terminated by "\r\n"
ignore 1 rows
(column_1, column_2, #variable1, #variable2, column_5)
set
Avg_Daily_Visitors = replace(#variable1,"\N",0),
pageviews = replace(#variable2,"\N", 0)
;
I create the following table and set default value of FLOAT as NULL because I want missing float value to be stored as NULL in the table. But the problem is after I set default as NULL in the table, all customerReviewAverage values become NULL.
The following are the code with which I created the table and loaded data.
CREATE TABLE Products(sku INTEGER, name VARCHAR(255), description TEXT,
regularPrice FLOAT,
customerReviewAverage FLOAT default NULL );
LOAD DATA LOCAL INFILE 'product.csv'
INTO TABLE Products
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(sku, name, #description, regularPrice, #customerReviewAverage)
SET description = IF(#description='',NULL,#description);
This is a sample of data in product.csv.
19658847,Glanzlichter - CD,,12.99,5.0
19658856,Glanzlichter - CD,,6.99,
19658865,Glanzlichter - CD,,8.99,
1965886,Beach Boys '69 - CASSETTE,,6.99,4.5
Later I found that
(sku, name, #description, regularPrice, #customerReviewAverage)
should be modified as
(sku, name, #description, regularPrice, customerReviewAverage)
Try this
(sku, name, #description, regularPrice, #customerReviewAverage)
SET
description = IF(#description='',NULL,#description),
customerReviewAverage = IF(#customerReviewAverage='',NULL,#customerReviewAverage);
Your "missing" values are an empty text in the CSV.
Since your mysql version isn't running this ok, you can do a separate update:
UPDATE Products SET customerReviewAverage = NULL WHERE customerReviewAverage = 0
I have created table as
CREATE TABLE TEST2
(Seq varchar2(255 CHAR),
ID varchar2(255 CHAR),
NAME VARCHAR2 (255 CHAR),
DOB TIMESTAMP(3)
);
my control file is
load data
infile 'C:\Users\sgujar\Documents\CDAR\test2.csv'
append into table TEST2
fields terminated by ","
(ID,
NAME,
DOB "TO_TIMESTAMP (:DOB, 'YYYY-MM-DD HH24:MI:SS.FF')",
seq"TEST2_seq.nextval"
)
I am not able to use sequence in sql loader.
Can you please help
Although not a particularly pretty solution, it does what you ask:
CREATE OR REPLACE
FUNCTION get_test2_seq RETURN INTEGER
IS
BEGIN
RETURN TEST2_seq.nextval;
END;
/
And then your control file would be
load data
infile 'C:\Users\sgujar\Documents\CDAR\test2.csv'
append into table TEST2
fields terminated by ","
(
ID,
NAME,
DOB "TO_TIMESTAMP (:DOB, 'YYYY-MM-DD HH24:MI:SS.FF')",
SEQ "get_test2_seq()"
)
This will work for sure
options (DIRECT=TRUE,readsize=4096000,bindsize=4096000,skip=1,errors=1,rows=50000)
LOAD DATA
CHARACTERSET AL32UTF8 LENGTH SEMANTICS CHARACTER
INFILE /path/test.csv'
BADFILE '/path/file.bad'
INSERT INTO TABLE test_table
FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY '"'TRAILING NULLCOLS
(
Col1 sequence(1,1),
Col2 constant "N",
)
I have the following data in the text file
Lat Lon Cls
-89.75 -179.75 EF
-89.75 -179.25 EF
-89.75 -178.75 EF
-89.75 -178.25 EF
-89.75 -177.75 EF
-89.75 -177.25 EF
-89.75 -176.75 EF
and i have to insert in in the mysql database in the following table
CREATE TABLE IF NOT EXISTS `jos_lat_log` (
`Lat` text NOT NULL,
`Lon` varchar(255) NOT NULL,
`climatefamily` varchar(255) NOT NULL
);
i am using the following query to insert
LOAD DATA LOCAL INFILE 'C:\\Koeppen-Geiger-ASCII.txt'
INTO TABLE `jos_lat_log` FIELDS TERMINATED BY '' LINES TERMINATED BY '\r\n';
the above query is working but it is inserting data in only one column but i have to insert it in all three columes
if your fields are seperated by a space,
TERMINATED BY ' '
with a space between quotes
Your fields in the file are terminated by a space ' ', but you have listed '' in your query. Just change FIELDS TERMINATED BY '' to FIELDS TERMINATED BY ' '