Hive - Complex regexp_replace - csv

Question

I'm not a specialist with regular expression and I'm facing issues using regexp_replace in Hive.
I would like to load a CSV file into Hive, which contains rows like that:
AAA,1234,BBB,,,"""CC,CCC""","""DDD""","""EE"EEE""",,
"""AAA""",1234,BBB,,,CCCC,"""DD,DD""",,"""FFFF""",
As you can see, the format isn't perfect
There are non-escaped commas into string fields
Some string fields are enclosed by """ (3 double-quotes)
There are non-escaped double-quotes into string fields
There are empty fields
When I try to import it into a Hive table, the columns are not well parsed because of the non-escaped commas.
So I imported the raw data as rows into a Hive table like this:
CREATE EXTERNAL TABLE MyRawTable
(
RAW_DATA STRING
)
STORED AS TEXTFILE
LOCATION '/path/to/hdfs/file'
And i'm trying to use the regexp_replace function to transform the rows:
Escape the commas, the double and simple quotes in the string fields
Not enclose string fields by double quotes
So data will look like that:
AAA,1234,BBB,,,CC\,CCC,DDD,EE\"EEE,,
AAA,1234,BBB,,,CCCC,DD\,DD,,FFFF,
I don't find the solution for this regex, any ideas? Thanks a lot!

Answer 1

Forget about the regexp, you don't need it. The commas aren't escaped, but they are surrounded by double-quotes. You can simply use the OpenCSVSerde :
CREATE EXTERNAL TABLE yourtable(foo int, bar string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
"separatorChar" = ",",
"quoteChar" = "\"",
"escapeChar" = "\""
)
LOCATION '/your/folder/containing/csv/files/';