We were trying to make calls in parallel to LLMs hosted in Bedrock, from a lambda layer (in python) only to discover that boto3 does not support async. Is there any workaround? I am looking into aiobotocore / aioboto3, but I do not find any example with Bedrock. Any hint appreciated and thank you very much! This is a minimal sample of the code I intended to use, but runs in sequence instead of parallel: nest_asyncio.apply() # async summaries async def _into_comment(segments: list[str]): bedrock = boto3.client( service_name="bedrock-runtime", aws_access_key_id=aws_access_key, aws_secret_access_key=aws_secret_key, aws_session_token=aws_session_token, region_name=aws_region ) async def sum_up(segment: str): body = json.dumps({ "max_tokens": 256, "messages": [{"role": "user", "content": f"Sumarize this: {segment}"}], "anthropic_version": "bedrock-2023-05-31" }) return bedrock.invoke_model(body=body, modelId=model_id) summaries = await asyncio.gather(*[sum_up(segment) for segment in segments]) return summaries summaries = asyncio.run(_into_comment(segments)) Continue reading...
