[Python] Python function to upload files to storage failing with 403 TooManyDevicesError

I have a python app running on an embedded linux device. The app runs async code that connects an iothub device client. I have a class called device_client_wrapper that uses a semaphore to ensure that only one function calls device client methods at a time, as that was causing me issues before.

I just added a new function to this class which is used to upload files from the device to azure blob storage. It was working fine for a while, and often works the first time, but then if I execute it too many times I get the following error:

ERROR upload_file Exception: Unexpected failure
ERROR Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 33, in handle_result
return await callback.completion()
File "/usr/lib/python3.10/site-packages/azure/iot/device/common/async_adapter.py", line 91, in completion
return await self.future
azure.iot.device.exceptions.ServiceError: HTTP operation returned: 403 TooManyDevicesError(Error: Forbidden)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/root/cogo_ig60/cloud/device_client_wrapper.py", line 99, in upload_file
blob_info = await self.device_client.get_storage_info_for_blob(blob_name)
File "/usr/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 571, in get_storage_info_for_blob
storage_info = await handle_result(callback)
File "/usr/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 57, in handle_result
raise exceptions.ClientError("Unexpected failure") from e
azure.iot.device.exceptions.ClientError: Unexpected failure


This is the function causing the error:

async def upload_file(self, path_to_file: str, blob_name: str):
"""Upload a file to Azure Blob Storage."""
try:
async with self.semaphore:
blob_info = await self.device_client.get_storage_info_for_blob(blob_name)
sas_url = "https://{}/{}/{}{}".format(
blob_info["hostName"],
blob_info["containerName"],
blob_info["blobName"],
blob_info["sasToken"]
)
logger.info(
f"Uploading file {path_to_file} to blob storage as {blob_name}")
with BlobClient.from_blob_url(sas_url) as blob_client:
with open(path_to_file, "rb") as f:
result = blob_client.upload_blob(f, overwrite=True)
return (True, result)
except FileNotFoundError as e:
# catch file not found and add an HTTP status code to return in notification to IoT Hub
logger.error(f'upload_file FileNotFoundError: {e}')
logger.error(traceback.format_exc())
e.status_code = 404
return (False, e)
except AzureError as e:
# catch Azure errors that might result from the upload operation
logger.error(f'upload_file AzureError: {e}')
logger.error(traceback.format_exc())
return (False, e)
except Exception as e:
# catch all other errors
logger.error(f'upload_file Exception: {e}')
logger.error(traceback.format_exc())
return (False, e)


It seems to be an issue with device_client.get_storage_info_for_blob. I know there are a max of 10 concurrent file uploads per device, but I am uploading 7 fiiles, then they finish uploading successfully before I try again. Even so, with the semaphore it should prevent any concurrent file uploads. That is unless I'm missing something about how my code is working.

Is something not being released properly, or not able to complete properly?

This is python 3.10.4 btw.

Continue reading...