[Python] How to deal with slightly different columns when scanning multiple csv files?

I have a folder containing thousands of csv files which I'd like to scan with PL lazy frame.

Scanning in fact works just fine, but when I try to fetch or collect the df, I get "ShapeError: unable to append to a dataframe of width 63 with a dataframe of width 67".

This means that there are some csv files containing more columns than the others. I checked that there are 4 unique sets of column names. Some csv files contains additional information compared to others.

Ideal outcome would be a dataframe containing all the possible columns and if there are some files that doesn't contain these specific "additional" columns, it would leave those blank.

Current code:

df = pl.scan_csv(r"path\*.csv", sep=";", infer_schema_length=0)
df.collect(streaming=True).write_parquet("new_file.parquet")


What would be a convenient way to deal with this?

Continue reading...