[Python] pandas/pyarrow ArrowTypeError: Unable to merge: Field on handcrafted partitioned...

I read the parquet file directly

pd.read_parquet(r'C:\Datasets\cn_data\dm\qmt\wqa_mfeatures\30m\year=2020\month=11\data.parquet')


No error will be reported.

But when I read the directory:

pd.read_parquet(r'C:\Datasets\cn_data\dm\qmt\wqa_mfeatures\30m\year=2020')


An error will be reported

ArrowTypeError: Unable to merge: Field month has incompatible types: int32 vs dictionary<values=int32, indices=int32, ordered=0>


This is because I handcrafted this partitioned path. It is important that I have to hand craft the partitioned path. I have 5000 item need transform and write to df.to_parquet(path, partition_cols=['year', 'month']) , and yes it would not overwrite existing files.

1. But if I only need rerun 300 item, it would preduce new files, I can't delete the data produce by last run.
2. With time goes, I need rerun transform function on new dates(year,month), I need new data overwrite old data.

pd.DataFrame.to_parquet doesn't support delete old files.

I just want to keep pd.read_parquet function working with handcraft paths, this can reduce many works on reimplement a similar stuff and refactor pd.read_parquet in many projects.

Continue reading...