I try to create a table but it always keeps the header I already have as a table row
DROP TABLE if exists schema.datasets;
CREATE TABLE schema.datasets
( YM_x String, OPER_Y_x String)
COMMENT 'from csv file'
ROW FORMAT DELIMITED FIELDS TERMINATED BY '073';
LOAD DATA LOCAL INPATH "datasets.csv"
OVERWRITE INTO TABLE schema.datasets;
tblproperties ("skip.header.line.count"="1");
I found out it is necessary to use tblproperties ("skip.header.line.count"="1") but it gives an error
and seems to reside in the wrong place
Try to bring the tblproperties while creating the table as below:
DROP TABLE if exists schema.datasets;
CREATE TABLE schema.datasets
(
YM_x String,
OPER_Y_x String
) COMMENT 'from csv file'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '073'
STORED AS TEXTFILE
tblproperties("skip.header.line.count"="1");
LOAD DATA LOCAL INPATH "datasets.csv" OVERWRITE INTO TABLE schema.datasets;
Related
I have a csv file which has contents like this.
"DepartmentID","Name","GroupName","ModifiedDate"
"1","Engineering","Research and Development","2008-04-30 00:00:00"
I have
create external table if not exists AdventureWorks2014.Department
(
DepartmentID smallint ,
Name string ,
GroupName string,
rate_code string,
ModifiedDate timestamp
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '","' lines terminated by '\n'
STORED AS TEXTFILE LOCATION 'wasb:///ds/Department' TBLPROPERTIES('skip.header.line.count'='1');`
And after loading the data
LOAD DATA INPATH 'wasb:///ds/Department.csv' INTO TABLE AdventureWorks2014.Department;
The data is not loaded.
select * from AdventureWorks2014.Department;
The above select returns nothing.
I think the double quotes around each fileds is the issue. Is there a way to load the data from such a file to hive tables, Without having to strip out the double quotes?
Try this (cellphone...)
create external table if not exists AdventureWorks2014.Department ( DepartmentID smallint , Name string , GroupName string, rate_code string, ModifiedDate timestamp )
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
STORED AS TEXTFILE
LOCATION 'wasb:///ds/Department'
** Limitation **
This SerDe treats all columns to be of type String. Even if you create a table with non-string column types using this SerDe, the DESCRIBE TABLE output would show string column type. The type information is retrieved from the SerDe. To convert columns to the desired type in a table, you can create a view over the table that does the CAST to the desired type.
https://cwiki.apache.org/confluence/display/Hive/CSV+Serde
FIELDS TERMINATED BY '","' is incorrect. Your fields are terminated by a , not ",". Change your DDL to FIELDS TERMINATED BY ','.
LOAD DATA LOCAL INPATH '/home/hadoop/hive/log_2013805_16210.log'into table_name
So I am very new in SQL and I am trying to create a table where I will later import a .csv file. In this table there is a time stamp column that I want to set it up to read mm/dd/yyyy hh:mi:ss, yet I've tried doing this:
create table Particle_counter_HiSam ( time_utc timestamp(m/d/Y hh:mi:ss),...
and i get this error
ERROR: syntax error at or near "m"
I just can't seem to figure this out.
Any help will do. Thanks!
Create the table as normal timestamp and use SET with STR_TO_DATE in load data infile as below.
-- table definition
create table Particle_counter_HiSam ( time_utc timestamp, ... );
-- load data
load data infile 'data.csv'
into table Particle_counter_HiSam
fields terminated BY ',' ESCAPED BY ""
lines terminated by '\r\n'
(#var1, c2, ....)
SET time_utc = STR_TO_DATE(#var1,'%m/%d/%Y %H:%i:%S');
if your creating a table for timestapm, just use this..
CREATE TABLE IF NOT EXIST 'Particle_counter_HiSam'
{
'date_log' timestamp NOT NULL,
}
hope this help..
i use below script for insert data to sql from textpad.
#!/bin/bash
mysql --utest -ptest test << EOF
LOAD DATA INFILE 'test.txt'
INTO TABLE content_delivery_process
FIELDS TERMINATED BY ',';
EOF
in my test file i have a format like,
cast , date , name , buy
i can insert but i need format like below,
S.NO | date | name | buy | cast
You can specify the columns you want to import:
From the MySQL Manual:
MySQL LOAD DATA INFILE
The following example loads all columns of the persondata table:
LOAD DATA INFILE 'persondata.txt' INTO TABLE persondata;
By default, when no column list is provided at the end of the LOAD
DATA INFILE statement, input lines are expected to contain a field for
each table column.
If you want to load only some of a table's columns, specify a column
list:
LOAD DATA INFILE 'persondata.txt' INTO TABLE persondata (col1,col2,...);
You must also specify a column list if the order of the fields in the
input file differs from the order of the columns in the table.
Otherwise, MySQL cannot tell how to match input fields with table
columns.
You would include "FIELDS TERMINATED BY '|';" at the end to import data delimited with a '|' symbol.
Hope this helps.
create table [YOUR TABLE] ( `S.NO` INT AUTO_INCREMENT, date DATETIME, name VARCHAR(50), buy VARCHAR(50), cast VARCHAR(50));
Load data local infile 'test.txt' ignore into table [YOUR TABLE] fields terminated by ',' lines terminated by '\n'(cast , date , name , buy);
I have a csv file with three columns- username, password and email. Now I have a table with columns wp_user, wp_password, wp_email among certain other columns. My question is how to I insert data from the csv file into my table, mapping the three columns and ignoring others?
If you'd simply import the csv file into a temporary table called mySourceTable, this would work:
INSERT INTO myTargetTable (wp_user, wp_password, wp_email)
SELECT username, password, email FROM mySourceTable
Edit2:
For mySQL, load the data into a temporary table using the following command:
load data local infile 'source.csv' into table mySourceTable fields terminated by ','
enclosed by '"'
lines terminated by '\n'
(username, password, email)
Edit:
Alternatively, you can use the csv file as an external table (for Oracle. Original question was tagged with plsql):
For your CSV file, you'll create a database directory to match your
already existing OS directory and grant your Oracle user READ and
WRITE access to it: SQL> CREATE OR REPLACE DIRECTORY my_data_dir as
'/my/data/dir/'; Directory created.
SQL> GRANT read, write ON DIRECTORY my_data_dir TO scott; Grant
succeeded.
You then create the external table definition
Create the external table definition:
CREATE TABLE t1 ( c1 NUMBER, c2 VARCHAR2(30) )
ORGANIZATION EXTERNAL
(
default directory
my_data_dir
access parameters
(
records delimited by newline fields terminated by ','
)
location
(
'report.csv'
)
);
The resulting table (t1) can then be used in the INSERT statement above.
http://www.orafaq.com/wiki/External_table
I am interested in importing a CSV file into a Hive table. The first field from the Hive table (ts) is of type BIGINT. After performing the following query, Hive ams_csv table is successfully created but the ts values are NULL.
CREATE EXTERNAL TABLE ams_csv (ts BIGINT, id STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/tmp/csvFilesDirectory';
I performed the same query but with the following modification and it worked:
CREATE EXTERNAL TABLE ams_csv (ts STRING, id STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/tmp/csvFilesDirectory';
I am not interested in having ts of type String. Does anyone know how to perform the cast. I thought it was implicit.
Many thanks!