[Python] Is it possible to have an atexit called back just before the main thread joins, but...

I am developing an automated trading package that utilizes a third-party api module provided by a broker.

This api module creates non-daemon threads, which can be shut down by explicitly calling api.cleanup() at the end of the user's program.

If my module broker wraps api so that broker is visible to the users but api is not, the users must call broker.cleanup() at the end of their programs. Otherwise their main programs will hang indefinitely due to those non-daemon threads.

If my robot module wraps broker such that only robot is visible to the users, then the users must call robot.cleanup() at the end of their programs.

Ultimately, an explicit cleanup() call is required.

To address the hanging issue, I use atexit.register(api.cleanup) in my broker module, aiming to relieve the users from manually cleaning up.

However, the atexit functions will never be called, as explained in this stackoverflow answer, because the Python interpreter waits the non-daemon threads to finish before exiting.

I found an alternative solution in this stackoverflow answer, by starting a thread to monitor the main thread and perform cleanup after the main thread has joined.

def cleanup_after_main_thread():
threading.main_thread().join()
api.cleanup()

threading.Thread(target=cleanup_after_main_thread).start()


However, this approach results in a RuntimeError: cannot schedule new futures after interpreter shutdown exception.

Exception in thread Thread-4 (cleanup_after_main_thread):
Traceback (most recent call last):
File "C:\Python311\Lib\threading.py", line 1045, in _bootstrap_inner
self.run()
File "C:\Python311\Lib\threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "C:\Projects\AutoTrade\broker.py", line 81, in cleanup_after_main_thread
api.cleanup()
File "C:\venv\AutoTrade\Lib\site-packages\BrokerABC\api.py", line 69, in cleanup
self._client.disconnect()
File "C:\venv\AutoTrade\Lib\site-packages\BrokerABC\soclient.py", line 104, in disconnect
self._message_service.disconnect()
File "C:\venv\AutoTrade\Lib\site-packages\solace\messaging\messaging_service.py", line 1321, in disconnect
_, error_info = self._session.disconnect()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\venv\AutoTrade\Lib\site-packages\solace\messaging\core\_solace_session.py", line 809, in disconnect
self.emit_publisher_event(_SolacePublisherEvent.PUBLISHER_DOWN, event_message)
File "C:\venv\AutoTrade\Lib\site-packages\solace\messaging\core\_solace_session.py", line 669, in emit_publisher_event
handler(error_info)
File "C:\venv\AutoTrade\Lib\site-packages\solace\messaging\publisher\_impl\_message_publisher.py", line 654, in on_publisher_down
self._on_publisher_down(error_info)
File "C:\venv\AutoTrade\Lib\site-packages\solace\messaging\publisher\_impl\_message_publisher.py", line 646, in _on_publisher_down
self._executor.submit(_handle_on_publisher_down, error_info)
File "C:\Python311\Lib\concurrent\futures\thread.py", line 169, in submit
raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown


IMHO, explicit cleanup at the end of the program works, while implicit cleanup after the main thread has joined results in a RuntimeError. I believe implicit cleanup just before the main thread finishes (but not after) would solve my problem.

Python interpreter has been popular for many years. I believe it provides a way to inject our functions at the end of the program, similar to atexit and sys.excepthook and Java's addShutdownHook().

Does it?

Notes:

1. 
   We cannot modify the provided third-party api module.
   
   
2. 
   I know that asking users to explicitly call robot.cleanup() is a workable solution, but I am inquiring about an implicit approach. All of my other modules have implicit cleanup. It's unfortunate that only this module does not.
   
   
3. 
   Using the __exit__ context method is also considered explicit, as the users must manage the scope. I know that __exit__ works, but it is not the solution I am asking about.
   
   
4. 
   I also tried using __del__, but it is never called because the program has not yet shut down. I am also aware that __del__ is unreliable.
   
   
5. 
   Currently, my workaround is to terminate all threads abruptly without graceful cleanup. This is only used to unlock a hung program and is not a suitable solution.
   

def brute_after_main_thread():
threading.main_thread().join()
os._exit(0)


Thank you in advance.

Continue reading...