My filename .dat file includes 1000 records in below format.
SIC_ID|NAME|CHANGED_DATE|LOGICALLY_DELETED
110|Wheat|31/10/2010 29:46:00|N
Table in which I want to feed content has few more columns. I wish to leave these columns blank as content is not there in .dat file.
Table Columns:
SIC_ID, NAME, CREATED_USER_ID ,CREATED_DATE ,CHANGED_USER_ID ,CHANGED_DATE,LOGICALLY_DELETED,RECORD_VERSION
My control file is as below:-
OPTIONS (DIRECT=TRUE,SKIP=1)
LOAD DATA CHARACTERSET WE8MSWIN1252
INFILE "mic_file.dat"
BADFILE "sql/mic_file.bad"
REPLACE
INTO TABLE SDS_SIC
FIELDS TERMINATED BY "|"
TRAILING NULLCOLS
(SIC_ID, NAME,
DATE "DD/MM/YYYY HH24:MI:SS" NULLIF (CHANGED_DATE=BLANKS),
LOGICALLY_DELETED)
After running SQL*Loader, I see below error:
Column Name Position Len Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
SIC_ID FIRST * | CHARACTER
NAME NEXT * | CHARACTER
CHANGED_DATE NEXT * | CHARACTER
LOGICALLY_DELETED NEXT * | CHARACTER
Record 1: Rejected - Error on table SDS_SIC, column CHANGED_DATE.
ORA-26041: DATETIME/INTERVAL datatype conversion error
last 2 lines of error is thrown multiple times. This is fixed now :)
Error 2: LOGICALLY_DELETED has only 2 possible values - Y or N.
Record 51: Rejected - Error on table SDS_SIC, column LOGICALLY_DELETED.
ORA-12899: value too large for column LOGICALLY_DELETED (actual: 2, maximum: 1)
Above error is displayed multiple times.
Remember that the control file column list is in the order fields are in the data file. Data is matched to the table columns by name. Your control file has the 3rd and 4th fields mapped to FILLER, that's why they are blank. FILLER only applies to a field in the data file you don't want.
You need something like this only in your column list section, the TRAILING NULLCOLS will handle the rest of the columns of the table:
(SIC_ID,
NAME,
CHANGED_DATE DATE "DD/MM/YYYY HH24:MI:SS" NULLIF (CHANGED_DATE=BLANKS),
LOGICALLY_DELETED
)
See this recent post which happens to describe the relationship by giving an example: Skipping data fields while loading delimited data using SQLLDR
You can go to the MySQL command line client and for inserting the values into the desired columns you should do the following:
like as you want to insert the value into
SIC_ID|NAME|CHANGED_DATE|LOGICALLY_ DELETED
And not in those extra columns. You should type:
insert into 'whatever the table name is'(SIC_ID,NAME,CHANGED_DATE,LOGICALLY_DELETED)
values(112,'wheat','31/10/2010 19:46:00',N);
Use single inverted commas only there where you have taken the property varchar of the columns
try it....it'll work
Related
Below is my table creation and a sample from my csv;
DROP TABLE IF EXISTS xxx.fbp;
CREATE TABLE IF NOT EXISTS xxx.fbp (id bigint, p_name string, h_name string, ufi int, city string, country string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;
74905,xxx,xyz,-5420642,City One,France
74993,xxx,zyx,-874432,City,Germany
75729,xxx,yzx,-1284248,City Two Long Name,France
I then load the data into a hive table with the following query:
LOAD DATA
INPATH '/user/xxx/hdfs_import/fbp.csv'
INTO TABLE xxx.fbp;
It seems that there is data leaking from the 5th csv "column" into the 6th column of the table. So, I'm seeing city data in my country column.
SELECT country, count(country) from xxx.fbp group by country
+---------+------+
| country | _c1 |
| Germany | 1143 |
| City | 1 |
+---------+------+
I'm not sure why city data is occasionally being imported to the country column. The csv is downloaded from Google Sheets and I've removed the header.
The reason could be your line termination is not '\n', Windows based tool add additional characters which creates issue. Also may be you have feilds using column separator creating this.
Solution:
1. Try print line which have issue by 'where country = City' clause, this will give you some idea how Hive created the record.
2. Try binary storage format to be 100% sure about data processed by Hive.
Hope it helps.
The issue was within the CSV itself. Some columns, such as p.name contained , in several fields. This would cause a line ending to end sooner than expected. I had to clean the data and remove all ,. After that, it imported correctly. Quickly done with python.
with open("fbp.csv") as infile, open("outfile.csv", "w") as outfile:
for line in infile:
outfile.write(line.replace(",", ""))
(MySQL) I have a table with 2 columns.
Table 2 column is a varbinary column.
I have txt file that has 2040 binary strings converted to numbers ( ie 000001 = 000001 , 000100 = 000004 etc).
I am tring to find a statement to insert the data into table. i tried
"LOAD DATA LOCAL INFILE 'C:/ProjectFolder/MySQLHex/Hex.txt' INTO TABLE testbinary
LINES TERMINATED BY '\n'
(#col1,#col2) set representation=#col2; " .
It inserted all rows as NULL values.
If i manually insert 1 row (insert statment), it works !! How do i load the txt file into 2 column of the table using a command ?
You're missing a FIELDS TERMINATED BY '=' in your statement. Therefore the whole line is treated as one column.
It inserted all rows as NULL values, because you load the whole line from your txt file into variables, and just the second column (which is NULL because your whole line is in variable #col1) is set to the variable #col2.
Since you don't do any transformations or whatever with your variables, those are completely unnecessary in this case. Just insert directly into the columns, without using variables.
I'm trying to import a XML file into a MySQL Table. In the XML file there is a timestamp in <CurrentTime> in the following format:
2016-01-26T09:52:19.3420655+01:00
This timstamp should go into the corresponding DATETIME CurrentTime column in my Table. So I did the following
LOAD XML INFILE 'xxx.xml'
INTO TABLE test.events
ROWS IDENTIFIED BY '<Event>'
SET CurrentTime = str_to_date(CurrentTime, '%Y-%m-%dT%H:%i:%s.%f');
But it quits with the error
Error Code: 1292. Incorrect datetime value: '2016-01-25T16:22:24.1840792+01:00' for column 'CurrentTime' at row 1
So it seems it doesn't convert the string at all. Why?
I think that error is thrown when the string value from the file is loaded directly to the column. The error is thrown before you get to the SET clause.
Here's an abbreviated example of how to use user-defined variables to pass the value of a field down to the SET, bypassing the assignment to the column.
Note that the columns _row and account_number are populated directly from the first two fields in the file. The later fields in the file are assigned to user-defined variables (identifiers beginning with #.
The SET clause evaluates the user-defined variables, and assigns the result of the expression to the actual column in the table.
In this example, the "dates" were formatted YYYYMMDD. I used the STR_TO_DATE() function to have that string converted to a proper DATE.
I abbreviated this sample somewhat, but it demonstrates the approach of reading field values into user-defined variables.
CREATE TABLE _import_water
(`_row` INT
,`account_number` VARCHAR(255)
,`total_due` DECIMAL(18,2)
,`end_date` DATE
,`start_date` DATE
,`ccf` DECIMAL(18,4)
)
LOAD DATA LOCAL INFILE '//server/share$/users/me/mydir/myfile.csv'
INTO TABLE _import_water
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(_row
,account_number
,#total_due
,#end_date
,#start_date
,#ccf
)
SET `total_due` = NULLIF(#total_due,'')
, `end_date` = STR_TO_DATE(#end_date,'%Y%m%d')
, `start_date` = STR_TO_DATE(#start_date,'%Y%m%d')
, `ccf` = NULLIF(#ccf,'')
Also, it doesn't look like there's any problem with your STR_TO_DATE, it seems to evaluate just fine.
testing...
SELECT STR_TO_DATE('2016-01-25T16:22:24.1840792+01:00','%Y-%m-%dT%H:%i:%s.%f') AS mydatetime
returns:
mydatetime
--------------------------
2016-01-25 16:22:24.184079
I have CSV file having two columns id_a and id_b, but I need to insert 4 more columns; ie. emp_sal_a, emp_sal_b, emp_dept_a, emp_dept_b using sqlldr. So my current control file looks like:
load data
infile '/home/.../employee.txt'
into table employee
fields terminated by ","
( id_a, id_b,
emp_sal_a ":id_a+1000", emp_sal_b "id_b+1000", emp_dept_a "10", emp_dept_b "20")
But I am getting error:
invalid binding variables
From MySQL Load Data Ref
note: search for the "(" character and it's the 35th instance of it on the page
User variables in the SET clause can be used in several ways. The following example uses the first input column directly for the value of t1.column1, and assigns the second input column to a user variable that is subjected to a division operation before being used for the value of t1.column2:
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(column1, #var1)
SET column2 = #var1/100;
#var1 is the name of a variable you want to run an operation on, and what you're doing is calling SET on column2 to be equal to #var1/100.
After changing the data type of a MySql column in order to store Twilio call ids (34 char strings), I try to manually change the data in that column with:
update calls
set incoming_Cid='CA9321a83241035b4c3d3e7a4f7aa6970d'
where id='1';
However I get an error which doesn't make sense seeing as the column's data type was properly modified?
| Level ||| Code | Message
| Warning | 1265 | Data truncated for column 'incoming_Cid' at row 1
Your problem is that at the moment your incoming_Cid column defined as CHAR(1) when it should be CHAR(34).
To fix this just issue this command to change your columns' length from 1 to 34
ALTER TABLE calls CHANGE incoming_Cid incoming_Cid CHAR(34);
Here is SQLFiddle demo
I had the same problem because of an table column which was defined as ENUM('x','y','z') and later on I was trying to save the value 'a' into this column, thus I got the mentioned error.
Solved by altering the table column definition and added value 'a' into the enum set.
By issuing this statement:
ALTER TABLES call MODIFY incoming_Cid CHAR;
... you omitted the length parameter. Your query was therefore equivalent to:
ALTER TABLE calls MODIFY incoming_Cid CHAR(1);
You must specify the field size for sizes larger than 1:
ALTER TABLE calls MODIFY incoming_Cid CHAR(34);
However I get an error which doesn't make sense seeing as the column's data type was properly modified?
| Level | Code | Msg | Warn | 12 | Data truncated for column 'incoming_Cid' at row 1
You can often get this message when you are doing something like the following:
REPLACE INTO table2 (SELECT * FROM table1);
Resulted in our case in the following error:
SQL Exception: Data truncated for column 'level' at row 1
The problem turned out to be column misalignment that resulted in a tinyint trying to be stored in a datetime field or vice versa.
In my case it was a table with an ENUM that accepts the days of the week as integers (0 to 6). When inserting the value 0 as an integer I got the error message "Data truncated for column ..." so to fix it I had to cast the integer to a string. So instead of:
$item->day = 0;
I had to do;
$item->day = (string) 0;
It looks silly to cast the zero like that but in my case it was in a Laravel factory, and I had to write it like this:
$factory->define(App\Schedule::class, function (Faker $faker) {
return [
'day' => (string) $faker->numberBetween(0, 6),
//
];
});
I had the same problem, with a database field with type "SET" which is an enum type.
I tried to add a value which was not in that list.
The value I tried to add had the decimal value 256, but the enum list only had 8 values.
1: 1 -> A
2: 2 -> B
3: 4 -> C
4: 8 -> D
5: 16 -> E
6: 32 -> F
7: 64 -> G
8: 128 -> H
So I just had to add the additional value to the field.
Reading this documentation entry helped me to understand the problem.
MySQL stores SET values numerically, with the low-order bit of the
stored value corresponding to the first set member. If you retrieve a
SET value in a numeric context, the value retrieved has bits set
corresponding to the set members that make up the column value. For
example, you can retrieve numeric values from a SET column like this:
mysql> SELECT set_col+0 FROM tbl_name; If a number is stored into a
If a number is stored into a SET column, the bits that are set in the
binary representation of the number determine the set members in the
column value. For a column specified as SET('a','b','c','d'), the
members have the following decimal and binary values.
SET Member Decimal Value Binary Value
'a' 1 0001
'b' 2 0010
'c' 4 0100
'd' 8 1000
If you assign a value of 9 to this column, that is 1001 in binary, so
the first and fourth SET value members 'a' and 'd' are selected and
the resulting value is 'a,d'.
when i first tried to import csv into mysql , i got the same error , and then i figured out mysql table i created doesn't have the character length of the importing csv field ,
so if it's the first time importing csv
its a good idea to give more character length .
label all fields as varchar or text , don't blend int or other values.
then you are good to go.
Check whether you are using the 'enum' datatype.If you are using 'enum' datatype then the items inside the enum datatype should be exactly match with the data which you are entering.Ex.
You are taking the enum datatype like this:
enum('book',stationery','others')
then when you are inserting the data into the database you have to do like this:
INSERT INTO database_name.table_name (column1,...columnn) VALUES().
THE value should include same items which you are mentioned within the bracket of enum datatype.
my issue was I used single quote instead of double quotes which add extra words in Boolean column
I Faced the same issue .In my case i was inserting a empty string for a numeric column.But by inserting a numeric value in string form for same column it got resolved e.g '12.56' --> numeric column will work but '' --> numeric column will give the above mentioned error.
Note: For numeric columns in MySql we can pass values in double quotes also .
In some cases this can be result of wrong type of input. For example, you have a column decimal(10,2) and the input isn't sanitized and is 7,7 but should be 7.7.