I have a FastAPI server where upon a HTTP GET (/get_documents/), there are 2 bottlenecks to returning a response fast: there is a call to another server via a client ("get vectors"), then there is a call to an LLM (in process memory) to generate text. Both are very slow. I'm thinking that the "get vectors" call to the other server could be put behind an await, so that the CPU is freed while waiting for the external call to come back. However, I don't think the call to the LLM in process would benefit from being behind an await, since it does require to complete on the CPU to answer the query. Further, I am not sure if I should make the /get_documents/ async for FastAPI. I keep reading about this, but get lost since there is talk of "threads" but python has the GIL... What would be the best strategy for the external call (via client) and the call to the LLM, as well as that /get_documents/ that runs in FastAPI? I'm hoping for a very simple solution, if possible. Continue reading...
