[Python] FastAPI becomes unresponsive while running partition_pdf from unstructured in a...

I have a FastAPI application where one of the API endpoints needs to parse PDF files using unstructured.partition_pdf. Since the parsing is CPU-heavy, I want to run it in the background so that the main FastAPI server remains responsive to other requests.

I'm currently offloading the parsing using the multiprocessing module like this:

from multiprocessing import Process
process = Process(target=run_file_parsing, args=(request.model_dump()))
process. Daemon = False
process.start()


And inside run_file_parsing, I do:

def run_file_parsing(request: dict):
try:
import asyncio
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
parsing_request = ParsingRequest(**parsing_request)
parsing = Parsing()
print(" Running ingestion.parse_files...")
results = loop.run_until_complete(parsing.parse_files(
parsing_request))
print("Parsing completed", results)


My parsing logic internally calls:

from unstructured.partition.pdf import partition_pdf
partition_pdf(
file=file,
strategy='hi_res',
chunking_strategy=strategy,
max_characters=max_characters,
# overlap=overlap,
combine_text_under_n_chars=50,
infer_table_structure=True
)


The Problem: Whenever partition_pdf starts processing the file, all FastAPI endpoints become unresponsive, even though the parsing runs in a separate Process. Other API requests just hang until the parsing is done. This happens even when running locally.

Things I've Tried:

• Verified that Process is used and not Thread (to avoid GIL contention).
• Tried using ProcessPoolExecutor — same result.
• Tried adding os.nice(10) to lower the priority — no improvement.
• Logging confirms that the parsing runs in a separate process.

My Objective: I want partition_pdf to run in the background (in its own process), without affecting FastAPI’s responsiveness. Other endpoints should respond to traffic normally during this time.

Question: Why does partition_pdf processing in a child process still make FastAPI unresponsive, and what changes should I make to truly isolate the parsing and ensure my API remains responsive?

Continue reading...