[Python] How to optimize for memory when querying and saving large dataframe to S3

I'm trying to query to Athena using boto3, getting a value from the dataframe, then saving the dataframe to S3.

from io import StringIO
import boto3
import awswrangler as wr

region = ""
access_key = ""
secret_key = ""
database = ""

s3 = boto3.resource(service_name='s3',
region_name=region,
aws_access_key_id = access_key,
aws_secret_access_key = secret_key)

sql = "SELECT * FROM tbl"
boto3.setup_default_session(region_name=region,aws_access_key_id=access_key, aws_secret_access_key=secret_key)
df = wr.athena.read_sql_query(sql=sql, database=database)
values = df['value'].unique() # get list of values from dataframe
csv_buffer = StringIO()
df.to_csv(csv_buffer, index=False)
s3.Object(s3_bucket_name, "s3://path/to/folder/values.csv").put(Body=csv_buffer.getvalue())


What would be a more efficient way to read the dataframe + save to S3 for large datasets? I have already tried optimizing by using awswrangler instead of pd.read_csv() which helped at first, but when the dataframe is larger > 700MB, it is causing memory issues.

Continue reading...