[Python] polars `write_parquet` to S3 with partitions makes local copy

I am observing an unintuitive behavior when writing to S3 from polars with specifying partitions. Namely,

• with use_pyarrow=False a copy of the files is created locally before uploading, and not cleanup up afterwards automatically
• with use_pyarrow=True I am getting Uploading to <file> FAILED with error When initiating multiple part upload for key <directory> in bucket <bucket>: AWS Error ACCESS_DENIED during CreateMultipartUpload operation: Anonymous users cannot initiate multipart uploads. Please authenticate.

Here is a repro:

session = ... # boto3 session
credentials = session.get_credentials()
storage_options = {
"aws_access_key_id": credentials.access_key,
"aws_secret_access_key": credentials.secret_key,
"aws_session_token": credentials.token,
"aws_region": "eu-west-1",
}
destination = f"s3://{bucket}/{directory}"

data_frame.write_parquet(
destination,
partition_by=partitions,
storage_options=storage_options,
use_pyarrow = ... # True | False, different behavior
)


I wonder if this is expected and/or I am missing something obvious

Continue reading...