[Python] How to Optimize Asynchronous Video Upload to S3 Python?

I'm making requests to various API endpoints that contain large mp4 files (>100 MB) and want to populate them to a S3 bucket. I concluded this as a I/O bound task due to the handling of API requests + streaming the video content to upload, which is why I chose to use asynchronous programming. My current code implementation isn't efficient enough when it comes to uploading to S3 however. How I approached it was by iterating over each chunk of the video first, then upload the entire thing (as shown in the code below), but realized that took a few minutes per video. I then tried uploading each iterated chunk similar to this implementation, but didn't see any improvements. I'm pretty sure this is a issue with how I'm handling each chunk of the video stream, but haven't found many solutions out there to help. I can load the video to my local directory with the each video showing and processing across tasks, but I noticed on S3 it looks like the videos are being processed one at a time. Is aioboto3 the correct solution to asynchronously write video files to S3? Is this an issue with how I'm utilizing aioboto3 with asynchronous programming? This is the example code:

import asyncio
import httpx
from botocore.exceptions import ClientError
import aioboto3
from boto3.s3.transfer import TransferConfig
from typing import AsyncIterator
import io


async def async_upload(stream: AsyncIterator[bytes],
destination_key: str,
config: TransferConfig) -> None:
"""
Asynchronously uploads video files via `aioboto3`

Args:
stream (AsyncIterator[bytes]): The video stream broken up into iterable chunks
destination_key (str): The desired destination for the video file
config (TransferConfig): Configuration for using threading
"""

session = aioboto3.Session()

async with session.client('s3') as s3:
buffer = io.BytesIO()
async for chunk in stream:
buffer.write(chunk)

buffer.seek(0)
await s3.upload_fileobj(buffer, bucket_name, destination_key, Config=config)


async def upload_video():
# Example url
url = 'https://api.example.com/somevideo.mp4'

async with httpx.AsyncClient(timeout=httpx.Timeout(20, read=10)) as request:
# Stream to lazily load to memory for faster times
async with request.stream("GET", url) as response:
if response.status_code == 200:
try:
s3_key = f'{folder_name}/{file_name}'
config = TransferConfig(use_threads=True)
await async_upload(response.aiter_bytes(chunk_size=100*1024*1024), s3_key, config) # 100MB chunks

except ClientError as e:
print('Upload Failed', e)
else:
print('Bad Response', response.status_code)

if __name__ == '__main__':

bucket_name = 'Test-Bucket'
folder_name = 'Test'
file_name = 'Video.MP4'

asyncio.run(upload_video())

Continue reading...