[Python] Unable to upload multiple files in s3 bucket

I am uploading image files in s3 bucket. I am using aioboto3 and aiofiles packages, and also created a virtual environment. But I am getting this error again and again.

1. Initializing the session client

async def upload_file_to_s3(session, bucket_name, folder_name, file_uuid, base64_data, mime_type):
file_extension = mimetypes.guess_extension(mime_type) or ""
object_name = f"{folder_name}/{file_uuid}{file_extension}"

try:
file_data = base64.b64decode(base64_data)

async with session.client('s3') as s3_client:
await s3_client.put_object(Bucket=bucket_name, Key=object_name, Body=file_data, ContentType=mime_type)

file_url = f"https://{bucket_name}.s3.{s3_client.meta.region_name}.amazonaws.com/{object_name}"
return file_url
except Exception as e:
print(f"An error occurred while uploading {file_uuid}: {e}")


2. Prepare image base 64 data and create tasks for s3 uplaods

async def main_s3_file_uploader(files_to_upload):
session = aioboto3.Session(
aws_access_key_id=aws_creds["aws_access_key_id"],
aws_secret_access_key=aws_creds["aws_secret_access_key"],
region_name=aws_creds["region_name"]
)

tasks = []
results = {}

for file_uuid, base64_data in files_to_upload.items():
mime_type = "image/jpeg"
if base64_data.startswith("data:image/png;base64,"):
mime_type = "image/png"
elif base64_data.startswith("data:image/jpeg;base64,"):
mime_type = "image/jpeg"
elif base64_data.startswith("data:image/gif;base64,"):
mime_type = "image/gif"

base64_data_clean = base64_data.split(",")[-1]

tasks.append(
(file_uuid, upload_file_to_s3(session, bucket_name, folder_name, file_uuid, base64_data_clean, mime_type))
)

for file_uuid, task in tasks:
result = await task
if result:
results[file_uuid] = result

return results


Error Message:

RuntimeError Traceback (most recent call last)
Cell In[1], line 91
89 search_results = read_search_results("Extra/save_search_results.json")
90 files_to_upload = prepare_img_data_s3_upload(search_results)
---> 91 results = asyncio.run(main_s3_file_uploader(files_to_upload))
92 print("********** Upload results **********")
93 print(results)

File /usr/lib/python3.10/asyncio/runners.py:33, in run(main, debug)
9 """Execute the coroutine and return the result.
10
11 This function runs the passed coroutine, taking care of
(...)
30 asyncio.run(main())
31 """
32 if events._get_running_loop() is not None:
---> 33 raise RuntimeError(
34 "asyncio.run() cannot be called from a running event loop")
36 if not coroutines.iscoroutine(main):
37 raise ValueError("a coroutine was expected, got {!r}".format(main))

RuntimeError: asyncio.run() cannot be called from a running event loop


I have tried changing the package versions, and used the latest ones, but got the same error

aioboto3==13.3.0 aiofiles==24.1.0

Edit:

I have noticed something odd, when I am running the same code in .py file its working fine. But when I am executing it in a .ipynb notebook its throwing the same error.

Output from .py file (note actual uuid and s3 links modified for Data Privacy)

python3 s3_upload.py
{'file_id_1': 's3_link_1.jpg', 'file_id_2': 's3_link_2.jpg', 'file_id_4': 's3_link_4.jpg', 'file_id_5': 's3_link_5.jpg'}

Continue reading...