How do you set existing_data_behavior in pyarrow? - pyarrow

Question

I'm getting this error. How do I change the behavior when writing a dataset (write_dataset)
pyarrow.lib.ArrowInvalid: Could not write to <my-output-dir> as the directory is not empty and existing_data_behavior is to error

Answer 1

Update: If you are using exactly version 6.0.0 then this was a bug (see below). If you are using a version >= 6.0.1 then you can specify it as part of the write_dataset call:
import pyarrow as pa
import pyarrow.dataset as ds
tab = pa.Table.from_pydict({"x": [1, 2, 3], "y": ["x", "y", "z"]})
partitioning = ds.partitioning(schema=pa.schema([pa.field('y', pa.utf8())]), flavor='hive')
ds.write_dataset(tab, '/tmp/foo_dataset', format='parquet', partitioning=partitioning)
# This write would fail because data exists and the default
# is to not allow a potential overwrite
ds.write_dataset(tab, '/tmp/foo_dataset', format='parquet', partitioning=partitioning)
# By specifying existing_data_behavior we can change that
# default to return to the previous behavior
ds.write_dataset(tab, '/tmp/foo_dataset', format='parquet', partitioning=partitioning, existing_data_behavior='overwrite_or_ignore')
Legacy 6.0.0 Answer
This is unfortunately a bug: https://issues.apache.org/jira/browse/ARROW-14620
The default behavior changed in 6.0.0 so that the write_dataset method will not proceed if data exists in the destination directory. The flag to override this behavior did not get included in the python bindings.
Workarounds are to use an older version or delete all files in the directory first.