I have a requirement to load data from delimited file to Oracle 12c.
My Ctl file looks like below :
OPTIONS (skip=2,rows=1000, bindsize=100000, readsize=100000, discardmax=1, silent=header,feedback)
load data
CHARACTERSET UTF8
insert into table XYZ
WHEN FIELD2 <> ''
fields terminated by '^' optionally enclosed by ','
trailing nullcols
(
FIELD1 filler,
FIELD2,
FIELD3,
MODIFIED_DATE "SYSDATE"
)
In my data file, i am expecting atleast one row with NULL for FIELD2 & FIELD3.
Assume my data file is :
ABC^DEF^GHI
ABC^^
My second record gets discarded as intended , with the below message :
1 Row successfully loaded.
0 Rows not loaded due to data errors.
1 Row not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
However, sqlldr exit code remains 2 , causing my script to fail.
Is there a way to load the valid records (rows with FIELD2 as not null) & gracefully success exit ?
Any help is greatly appreciated.
Thanks.
Any WHEN condition that fails will cause sqlldr to exit non-zero, as you have discovered.
You will need to either:
Pre-process the file and remove those lines with NULL fields before loading
Load all records into your table then delete those with the NULL
second column
Load all records into a staging table then select the rows you want
into the main table
Use an external table and select the rows you want into the main
table
Each method has its pros and cons so you'll have to do some studying and see which will work best for your situation.
Related
LOAD DATA INFILE 'thefile.csv'
INTO TABLE word
FIELDS TERMINATED BY ';'
IGNORE 1 LINES
SET id = NULL;
I am a bit lost of what to do here, I thought this query would work but it doesn't. I get the error about id, which is why I thought set id = NULL would make it good but no. The point of my id row (which is AUTO_INCREMENT) is that I don't need to specify the ID. so.. ehm. Any help would be greatly appreciated
Incorrect integer value: '' for column 'id' at row 1
My CSV file content:
id;meaning;japanese;kanji;kana;romaji;visible;featured;image;image_author;imgauthor_link;extra
;pants;パンツ;;パンツ;pantsu;;;;;;
You must specify all column names except the id column:
LOAD DATA INFILE 'thefile.csv'
INTO TABLE word
FIELDS TERMINATED BY ';'
IGNORE 1 LINES
(col1,col2,col3);
I assumed your table has (id,col1,col2,col3) columns.
You can use a two step load:
1) Load the data into a table;
2) Use INSERT INTO ... SELECT ... FROM to get your data in your table
Example:
CREATE TABLE baseData (
All your column definitions, but not your id column
);
INSERT INTO (all your columns except the id column) SELECT * FROM baseData;
Alternative option:
Create your table with the id as last column, then the loading with LOAD DATA INFILE works
(MySQL) I have a table with 2 columns.
Table 2 column is a varbinary column.
I have txt file that has 2040 binary strings converted to numbers ( ie 000001 = 000001 , 000100 = 000004 etc).
I am tring to find a statement to insert the data into table. i tried
"LOAD DATA LOCAL INFILE 'C:/ProjectFolder/MySQLHex/Hex.txt' INTO TABLE testbinary
LINES TERMINATED BY '\n'
(#col1,#col2) set representation=#col2; " .
It inserted all rows as NULL values.
If i manually insert 1 row (insert statment), it works !! How do i load the txt file into 2 column of the table using a command ?
You're missing a FIELDS TERMINATED BY '=' in your statement. Therefore the whole line is treated as one column.
It inserted all rows as NULL values, because you load the whole line from your txt file into variables, and just the second column (which is NULL because your whole line is in variable #col1) is set to the variable #col2.
Since you don't do any transformations or whatever with your variables, those are completely unnecessary in this case. Just insert directly into the columns, without using variables.
I have CSV file having two columns id_a and id_b, but I need to insert 4 more columns; ie. emp_sal_a, emp_sal_b, emp_dept_a, emp_dept_b using sqlldr. So my current control file looks like:
load data
infile '/home/.../employee.txt'
into table employee
fields terminated by ","
( id_a, id_b,
emp_sal_a ":id_a+1000", emp_sal_b "id_b+1000", emp_dept_a "10", emp_dept_b "20")
But I am getting error:
invalid binding variables
From MySQL Load Data Ref
note: search for the "(" character and it's the 35th instance of it on the page
User variables in the SET clause can be used in several ways. The following example uses the first input column directly for the value of t1.column1, and assigns the second input column to a user variable that is subjected to a division operation before being used for the value of t1.column2:
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(column1, #var1)
SET column2 = #var1/100;
#var1 is the name of a variable you want to run an operation on, and what you're doing is calling SET on column2 to be equal to #var1/100.
I'm learning MySQL and PHP (running XAMPP and also using HeidiSQL) but have a live project for work that I'm trying to use it instead of the gazillion spreadsheets in which the information is currently located.
I want to import 1,000+ rows into a table (tbl_searches) where one of the columns is a string (contract_no). Information not in the the spreadsheet required by tbl_searches includes search_id (PK and is AUTO_INCREMENT) and contract_id. So the only field I am really missing is contract_id. I have a table (tbl_contracts) that contains contract_id and contract_no. So I think I can have the import use the string contract_no to reference that table to grab the contract_id for the contract_no, but I don't know how.
[EDIT] I forgot to mention I have successfully imported the info using HeidiSQL after I exported the tbl_contracts to Excel and then used it the Excel VLOOKUP function but that ended up yielding incorrect data somehow.
You can do it like this
LOAD DATA LOCAL INFILE '/path/to/your/file.csv'
INTO TABLE table1
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n' -- or '\r\n' if the file has been prepared on Windows
(#field1, #contract_no, #field2, #field3,...)
SET column1 = #field1,
contract_id = (SELECT contract_id
FROM tbl_contracts
WHERE contract_no = #contract_no
LIMIT 1),
column2 = #field2,
column3 = #field3
...
try something like this: (I am assuming that you have data in tbl_contracts)
<?php
$handle = fopen("data_for_table_searches.csv", "r");
while (($data = fgetcsv($handle,",")) !== FALSE) { // get CSV data from you file
$contract_id = query("SELECT contract_id FROM tbl_contracts WHERE contract_number = " . $data[<row for contract number>]); // whatever is the equivalent in heidi SQL, to get contract id
query("INSERT INTO tbl_searches values($contract_id, data[0], data[1], data[2],...)"); // whatever is the equivalent in heidi SQL, insert data, including contract id into tbl_searches
}
fclose($handle);
?>
Thanks for everyone's input. peterm's guidance helped me get the data imported. Rahul, I should have mentioned that I was not using PHP for this task, but rather just trying to get the data into the tables using HeidiSQL. user4035 asked for more detail and so that's here too.
I have three tables in the database.
tbl_status has two fields, status_ID (AUTO_INCREMENT) and status_name.
tbl_contracts has two columns, contract_ID (AUTO_INCREMENT) and contract_no (a string).
The last table (tbl_searches) will be the active(?) table in that this is where the users' actions will be recorded.
The first two of these tables were easily populated. tbl_status has 11 rows that will describe the status of the contract and these were just typed into an Excel spreadsheet and imported via CSV through HeidiSQL.
For the second table I had 1,000+ "contracts" to import and so I left the first column in Excel blank and the second column containing the string of the contract and imported them the same way.
The third table has seven fields: search_id (AUTO_INCREMENT), contract_id, contract_no, status_id, notes, initials and search_date (I forgot about that one until just now).
I wanted to insert the spreadsheet that had the search information on it into tbl_searches. It has the contract_no, but not the contract_id. I needed to insert the rows and have the query grab the contract_id from tbl_contracts. It took me a bit to get it right without errors and some unexpected results. (The following query omits the need for search_date.)
LOAD DATA LOCAL INFILE '\\\\PATH\\PATH\\PATH\\PATH\\FILENAME.csv'
INTO TABLE `hoa_work`.`tbl_searches`
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' LINES TERMINATED BY '\r\n'
IGNORE 1 LINES --because the first row of the CSV has column headers
(#search_id, #contract_id, #contract_no, #status_id, #notes, #initials)
SET
search_id = NULL, --is an AUTO_INCREMENT field
contract_id = (SELECT contract_id
FROM tbl_contracts
WHERE contract_no = #contract_no
LIMIT 1),
contract_no = #contract_no,
status_id = #status_id,
notes = #notes,
initials = #initials;
/* Affected rows: 1,011 Found rows: 0 Warnings: 0 Duration for 1 query: 0.406 sec. */
I learned here that the #blah are user variables. If I run the following query it will tell me how the variable is defined. Since I was inserting 1,000+ rows from the CSV file it gave me the answer for the last row that it inserted.
SELECT #contract_no
If you have any suggested improvements on the way I ultimately wrote the query please do tell me.
-Matt
I am trying to load a .csv file that has 5 columns into a table that has the same corresponding columns plus a PK. Dates are data type DATE and all others are Varchar(), except PK.
Here is my load data Import:
LOAD DATA INFILE 'C:\Events_Upload.csv'
INTO TABLE db.events
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
Ignore 1 lines
(
Event_Name,
Event_Handle,
Create_Date,
Retire_Date,
Event_Desc
)
The error is :
Error Code: 1292. Incorrect date value: '' for column 'Retire_Date' at row 1
Row one in CSV looks like:
Time to Send Survey,SurvESend,2013-04-10,,Time for the system to send out the Esurveys
How can I make this field NULL and not blank for uploading from CSV?
There is nothing as null in CSV file. What you can do is may be provide some default value for the missing column and replace it with null by the query after insertion into the table.
Or you can have a look at skipping-empty-csv-objects
check this Export null to .csv and https://superuser.com/questions/390031/how-to-write-null-into-csv-from-excel-for-blank-fields