[Python] Why httpx timeout not working with stream response

I use httpx to make request to my LLM. Here is the code.

import httpx
import logging
import json

from httpx import TimeoutException


try:
content = ''
logging.info(f"request payload:{payload}")
async with httpx.AsyncClient(timeout=LLM_TIMEOUT) as client:
async with client.stream("POST", MODEL_URL, headers=HEADERS, json=payload,
timeout=LLM_TIMEOUT) as response:
if response.status_code != 200:
ans = await response.aread()
logging.error(f"response.aread() error")
raise Exception(f"Failed to generate completion stream: {ans}")
async for line in response.aiter_lines():
if line.startswith("data:"):
data_str = line[5:]
if data_str.strip() == "[DONE]":
break
try:
chunk_data = json.loads(data_str)
if chunk_data['choices']:
content += chunk_data["choices"][0]["delta"].get("content")
except json.JSONDecodeError:
logging.error("json parse error:", data_str)
logging.info(f"request response:{content}")
except TimeoutException as te:
logging.warning("request timeout")


Here the LLM_TIMEOUT is 60. However, I can see from my log that there exists some requests that costs more than 60 seconds. Anyone can help me solve this? Thanks in advance.

Continue reading...