[Python] Should I use threading for a CPU-intense operation that is choking my service

I am a maintainer of an ML service based on FastAPI and we I have CPU intense code that runs on main thread and with high CPU utilisation when a request heads down this path (I've seen occasional throttling too of cycles too).

def _svd_with_fallback(self, matrix: np.ndarray) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
try:
return np_singular_value_decomposition(matrix, full_matrices=False)
...


What are approaches to solving this problem? We have spun up more gunicorn servers already, would advice here be to submit the workload to a separate thread?

Anyone have any best-practice tips?

Continue reading...