I have a FastAPI endpoint (/generateStreamer) that generates responses from an LLM model. I want to stream the output so users can see the text as it’s being generated, rather than waiting for the full response. Currently, I'm using TextIteratorStreamer from the transformers library and FastAPI's StreamingResponse, which works, but when I test with curl, the response arrives sentence by sentence instead of word by word. I've read several SO topics on similar issues, but none offer me a solution to this problem. I would appreciate any advice on how to stream the response more granularly, ideally word by word. @router.post( "/generateStreamer" ) @api_version(1) async def generateStreamer(features: InputFeatures, request: Request): print("GenerateStreamer endpoint was called, generating response...") tokenizer = request.app.state.tokenizer model = request.app.state.model messages = [ {"role": "system", "content": features.prompt.strip()}, {"role": "user", "content": features.text.strip()} ] prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = tokenizer(prompt, return_tensors="pt").to(model.device) streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) def generate(): torch.set_grad_enabled(False) model.generate( inputs['input_ids'], streamer=streamer, max_new_tokens=512, num_return_sequences=1, do_sample=True, temperature=0.7, repetition_penalty=1, pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id, top_p=1, top_k=50 ) thread = Thread(target=generate) thread.start() return StreamingResponse(streamer, media_type="text/plain") Continue reading...
