Cut when csv file has commas in cell linux - mysql

Question

I have IMDB data in csv format. Here is a snapshot.
[root#jamatney IMDB]# head IMDBMovie.txt
id,name,year,rank
0,#28 (2002),2002,
1,#7 Train: An Immigrant Journey, The (2000),2000,
2,$ (1971),1971,6.4000000000000004
3,$1000 Reward (1913),1913,
4,$1000 Reward (1915),1915,
5,$1000 Reward (1923),1923,
6,$1,000,000 Duck (1971),1971,5
7,$1,000,000 Reward, The (1920),1920,
8,$10,000 Under a Pillow (1921),1921,
I'd like to import this data into a MySQL database. However there are commas present in the name cells. This prevents me from loading the data into the database correctly, as my loading query is,
mysql> LOAD DATA LOCAL INFILE 'IMDB/IMDBMovie.txt' INTO TABLE Movie FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES;
I've thought about using some combination of rev and cut to isolate the offending column, then find/replace the commas, but can't seem to get it to work. Was wondering if this is the right approach, or if there's a better way.

Answer 1

It looks like the first field and last two fields are unambiguous, so all you have to do is write a script to pull those out, and surround what remains in quotes. My bash-fu isn't quite good enough to get it done with rev and cut, but I was able to write a Python script to get it done. You can add an OPTIONALLY ENCLOSED BY clause to your LOAD DATA query.
f = open("IMDBMovie.txt")
print(next(f)) # header
for line in f:
fields = line.strip().split(",")
# Get unambiguous fields.
id = fields.pop(0)
rank = fields.pop(-1)
year = fields.pop(-1)
# Surround name with quotes.
name = '"{}"'.format(",".join(fields))
print("{},{},{},{}".format(id, name, year, rank))
On your test data, the output was
id,name,year,rank
0,"#28 (2002)",2002,
1,"#7 Train: An Immigrant Journey, The (2000)",2000,
2,"$ (1971)",1971,6.4000000000000004
3,"$1000 Reward (1913)",1913,
4,"$1000 Reward (1915)",1915,
5,"$1000 Reward (1923)",1923,
6,"$1,000,000 Duck (1971)",1971,5
7,"$1,000,000 Reward, The (1920)",1920,
8,"$10,000 Under a Pillow (1921)",1921,

Answer 2

This is too long for a comment.
Good luck. Your input file is in a lousy format. It is not really CSV. Here are two options:
(1) Open the file in Excel (or your favorite spreadsheet) and save it out with tab delimiters instead. Keep your fingers crossed that none of the fields have a tab. Or use another delimiter such as a pipe character.
(2) Load each row into a table with only one column, a big character string column. Then, parse the rows into their constituent fields (substring_index() can be very useful).