[Python] How to write polars dataframe to S3 with partition_by?

I am able to write a polars dataframe to S3 when partition_by=None using the following code:

import os
import s3fs
import polars as pl

df = pl.DataFrame({'a': [1, 2, 3, 4], 'b': ['left', 'left', 'right', 'right']})

os.environ['AWS_CA_BUNDLE'] = r'C:\my\path\to\cert.pem'

fs = s3fs.S3FileSystem(
key='my-key',
secret='my-secret',
endpoint_url='https://my-endpoint-url.net'
)

with fs.open('s3://my-bucket-name/my/path/to/file', mode='wb') as f:
df.write_parquet(f)


However, when I set partition_by='b' using the following code:

import os
import s3fs
import polars as pl

df = pl.DataFrame({'a': [1, 2, 3, 4], 'b': ['left', 'left', 'right', 'right']})

os.environ['AWS_CA_BUNDLE'] = r'C:\my\path\to\cert.pem'

fs = s3fs.S3FileSystem(
key='my-key',
secret='my-secret',
endpoint_url='https://my-endpoint-url.net'
)

with fs.open('s3://my-bucket-name/my/path/to/file', mode='wb') as f:
df.write_parquet(f, partition_by='b')


I get the following error:

TypeError: 'S3File' object cannot be converted to 'PyString'


How do I write polars dataframe to S3 with partition_by='b'?

My Python environment:

python 3.9.7
polars 1.26.0
s3fs 2025.3.2
aiobotocore 2.17.0
botocore 1.35.93
boto3 1.35.93

Continue reading...