Cassandra invalid row length on copy (import) - csv

Question

I'm trying to import a large dataset (this one https://www.kaggle.com/secareanualin/football-events/data) into cassandra but I'm stuck. I created the table with the following command :
create table test.football_event(id_odsp text, id_event text, sort_order text, time text, text text, event_type text, event_type2 text, side text, event_team text, opponent text, player text, player2 text, player_in text, player_out text, shot_place text, shot_outcome text, is_goal text, location text, bodypart text, assist_method text, situation text, fast_break text, primary key(id_odsp));
This table matches the csv containing the data. When I try to import with this command
copy test.football_event(id_odsp, id_event, sort_order, time, text, event_type, event_type2, side, event_team, opponent, player, player2, player_in, player_out, shot_place, shot_outcome, is_goal, location, bodypart, assist_method, situation, fast_break) from '/path/to/events_import.csv' with delimiter = ',';
I'm getting the following error Failed to import XX rows: ParseError - Invalid row length 24 should be 23, given up without retries or same error with row length 23 should be 22. I assume that the data in the csv aren't perfect and that there are some errors so I increased the number of columns in my table to 24 but this didn't resolve the problem.
I was wondering if it didn't exist an option to manage the level of "strictness" during import but I didn't find anything about it. I would like an option that would allow me to fill up the entire table row when length is 24 or add one or two null in the last fields if row length is 23 or 22.
If it has any importance, I'm running cassandra on Linux Mint 18.1
Thanks in advance

Answer 1

Cassandra/Scylla are schema forced systems, the schema should include any required column. The copy command expects same number of elements fetched as specified in the columns part of the command.
In Cassandra/Scylla the copy command should create an error file on your loader node, the error file should include the rows that "created" the issue. you can review the wrong rows and decide if they are of interest for you, and remove/fix them.
It does not mean the other rows were not uploaded correctly. See example below:
The csv files looks as the following:
cat myfile.csv
id,col1,col2,col3,col4
1,bob,alice,charlie,david
2,bob,charlie,david,bob
3,alice,bob,david
4,david,bob,alice
cqlsh> create KEYSPACE myks WITH replication = {'class':'SimpleStrategy', 'replication_factor': 1};
cqlsh> USE myks ;
cqlsh:myks> create TABLE mytable (id int PRIMARY KEY,col1 text,col2 text,col3 text ,col4 text);
cqlsh> COPY myks.mytable (id, col1, col2, col3 , col4 ) FROM 'myfile.csv' WITH HEADER= true ;
Using 1 child processes
Starting copy of myks.mytable with columns [id, col1, col2, col3, col4].
Failed to import 2 rows: ParseError - Invalid row length 4 should be 5, given up without retries
Failed to process 2 rows;
failed rows written to import_myks_mytable.err
Processed: 4 rows; Rate: 7 rows/s; Avg. rate: 10 rows/s
4 rows imported from 1 files in 0.386 seconds (0 skipped).
cqlsh> SELECT * FROM myks.mytable ;
id | col1 | col2 | col3 | col4
----+------+---------+---------+-------
1 | bob | alice | charlie | david
2 | bob | charlie | david | bob
The Error file explains which rows have an issue:
cat import_myks_mytable.err
3,alice,bob,david
4,david,bob,alice