[Python] How to safely cancel a running Hugging Face text generation mid-inference without...

When running large models with Hugging Face's generate() function, if a user decides to cancel a request (e.g., UI "stop" button), the server has no way to interrupt an ongoing generate() call.

Currently, it looks like:

1. generate() blocks the Python thread until completion
2. No safe cancellation mechanism exists
3. For very large outputs, this can cause hanging issues

Example Scenario:

from transformers import pipeline

generator = pipeline('text-generation', model="TheBloke/Mistral-7B-Instruct-v0.1-GGUF", device=0)

# User sends a long prompt
output = generator("Tell me a very detailed history of the Roman empire...", max_new_tokens=1000)


If the user cancels after 2 seconds, there is no obvious way to stop generate().

My Question:

How can I implement a user-driven cancel mechanism for Hugging Face text generation that safely interrupts generate() mid-execution without crashing the server or leaking GPU memory?

Continue reading...