[Python] ArrowInvalid: GetFileInfo() yielded path which is outside base dir parquet

I have a parquet dataset stored in my S3 bucket with multiple partition files. I want to read it into my pandas dataframe, but am getting this ArrowInvalid error when I didn't before.

Occasionally, this data has been overwritten with some previous snapshot of pandas data like the following:

import pandas as pd # version 1.3.4
# pyarrow version 5.0

df.to_parquet(
f's3a://{bucket_and_prefix}',
storage_options={
"key" : os.getenv("AWS_ACCESS_KEY_ID"),
"secret" : os.getenv("AWS_SECRET_ACCESS_KEY"),
"client_kwargs": {
'verify' : os.getenv('AWS_CA_BUNDLE'),
'endpoint_url': 'https://prd-data.company.com/'
}
},
index=False
)


But when reading it with:

df = pd.read_parquet(
f"s3a://{bucket_and_prefix}",
storage_options={
"key" : os.getenv("AWS_ACCESS_KEY_ID"),
"secret" : os.getenv("AWS_SECRET_ACCESS_KEY"),
"client_kwargs": {
'verify' : os.getenv('AWS_CA_BUNDLE'),
'endpoint_url': 'https://prd-data.company.com/'
}
}
)


It fails with error:

ArrowInvalid: GetFileInfo() yielded path 'bucket/folder/data.parquet/year=2021/month=2/abcde.parquet', which is outside base dir 's3://bucket/folder/data.parquet'
​

Any idea why this ArrowInvalid error happens and how I can read the parquet data into pandas?

Continue reading...