[Python] OpenAI Azure chat.completions.create stalling without throwing exception

I have a fastapi endpoint which streams a response from Azure OpenAI:

@router.post("/message", dependencies=[Depends(validate_auth)])
def message(sessionID: str = Form(...), messages: str = Form(...), selectedModel: str = Form(...), file: UploadFile = Form(None)):
message_list = json.loads(messages)
return StreamingResponse(stream_chat(message_list, sessionID, selectedModel))

def stream_chat(messages: list[ChatCompletionMessageParam], session_id: str, selected_model: str) -> Generator[StreamingResponseChunk, None, None]:
print('entered stream_chat')
completion_stream = azure_openai_client.chat.completions.create(
model=selected_model,
messages=messages,
stream=True,
timeout=45,
**DEFAULT_MODEL_CONFIG
)
print('completion_stream created!')
#. . .



On occasion, azure_openai_client.chat.completions.create will infinitely stall, and cause every subsequent request to infinitely stall as well. Once the error occurs, every request reaches 'entered stream_chat' but not 'completion_stream created!', until I reset the FastAPI instance. No exceptions are raised, even after hitting the 45 second timeout. I haven't been able to recreate the bug consistently either. What on earth is going on?

Continue reading...