[Python] How do I terminate the process running a Celery task during a long-running Python...

I want to be able to terminate the process running a Celery task while it is running a long Python function within the task:

@shared_task()
def celery_task():
# I want to abort this task, interrupting this long function
long_function()
return "done!"


The AbortableTask class in Celery appears to not fit my use case (from the documentation):

@app.task(bind=True, base=AbortableTask)
def long_running_task(self):
results = []
for i in range(100):
# check after every 5 iterations...
# (or alternatively, check when some timer is due)
if not i % 5:
if self.is_aborted():
# respect aborted state, and terminate gracefully.
logger.warning('Task aborted')
return
value = do_something_expensive(i)
results.append(y)
logger.info('Task complete')
return results


This requires the task to have natural points to check if the task was aborted between shorter function calls, but the function that I am running is several minutes long, and I want to be able to terminate it.

The other option appears to be revoking the task, which the Celery documentation clearly advises against:

The terminate option is a last resort for administrators when a task is stuck. It’s not for terminating the task, it’s for terminating the process that’s executing the task, and that process may have already started processing another task at the point when the signal is sent, so for this reason you must never call this programmatically.
​

For further context, this is for a Flask application where I want the user to be able to stop the task that they submitted if they realized that they wanted to change any inputs, because the task takes multiple minutes to complete. Each user can only see the results of the latest task that they submitted. I understand that there would be no harm in letting the worker complete the task and simply allow the user to submit another task (current implementation), but my concern would be that workers would be occupied with unnecessary tasks that could be freed up if users were to submit multiple tasks.

Continue reading...