Hive external table pointing to CSV file with embedded double quotes - csv

Question

I am trying to create an external Hive table pointing to a CSV file.
My CSV file has a column(col2) that could have double quotes and comma as part of the column value.
Data in each column:
Col1 : 150
Col2 : BATWING, ABC "D " TEST DATA
Col3 : 300
Row in CSV:
150,"BATWING, ABC ""D "" TEST DATA",300
Create table DDL :
CREATE EXTERNAL TABLE test (
col1 INT,
col2 STRING,
col3 INT)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
ESCAPED BY '"'
LOCATION 's3://test-folder/test-file.csv'
When I query the table, I see null values in col3.
What am I missing here while creating the table? Any help is appreciated

Answer 1

Use OpenCSVSerde. Here is an example
Create table
CREATE TABLE bala (col1 int, col2 string, col3 int)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES(
"separatorChar" = ",", "escapeChar"='\"'
);
Load data
hive>LOAD DATA INPATH '/../test.csv' INTO TABLE bala
Loading data to table bala
Table testing.bala stats: [numFiles=1, totalSize=40]
OK
Time taken: 0.514 seconds
Check if it has loaded
hive> select * from bala;
OK
150 BATWING, ABC "D " TEST DATA 300
Time taken: 0.288 seconds, Fetched: 1 row(s)

Answer 2

Create hive external table:
DROP TABLE IF EXISTS ${hiveconf:dbnm}.tblnm ;
CREATE EXTERNAL TABLE ${hiveconf:dbnm}.tblnm (
C1 string,
C2 string
)
PARTITIONED BY (C3 string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
"separatorChar" = '|' (change it to your separator)
,"quoteChar" = '\"'
)
STORED AS TEXTFILE
LOCATION '/hdfspath'
--tblproperties ("skip.header.line.count"="1")
;
MSCK REPAIR TABLE ${hiveconf:dbnm}.tblnm;