[Python] what would be the best way to launch a detached process from a python process?

I have a fastapi aplication and I want to launch a another process with a python script in an non-blocking way:

process = subprocess.Popen(
["python","my_script.py"],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
stdin=subprocess.DEVNULL,
start_new_session=True # Detaches the process
)


Now, the problem is if that process fails it still lingers on as a zombie process. Only if I restart the fastapi aplication the process really dies. If I try to use os.fork in my_script.py to make sure I detach from the parent then a new process with new pid will be created and the original one will remain zombie.

The question is, from a system engineering point of view what is the most reliable way (and production suitable) to start/trigger python scripts from a fastapi aplication that can die properly if there is a problem ? I can not use celery because I'm trying to train machine learning models and this can take a lot of time and resources and I'm afraid celery workers are not suitable for that but please correct me if I'm wrong.

Continue reading...