[Python] Embedded Python (3.10) - Py_FinalizeEx hangs/deadlock on "threading._shutdown()"

I am embedding Python in a C++ application, and I think I have some confusion with PyGILState_Ensure/PyGILState_Release which leads eventually to Py_FinalizeEx to hang in threading._shutdown() (called by Py_FinalizeEx) while join()-ing the threads.

During initialization I am calling:

Py_InitializeEx(0); // Skip installing signal handlers
auto gil = PyGILState_Ensure();
// ... running Python code
PyGILState_Release(gil);


Whenever a thread uses Python (can be multiple C-threads), I am using pyscope at the beginning of the function:

#define pyscope() \
PyGILState_STATE gstate = PyGILState_Ensure(); \
utils::scope_guard sggstate([&]() \
{ \
PyGILState_Release(gstate); \
});


When I want to free python I am calling (from a C-Thread, not necessarily one who initialized Python):

PyGILState_STATE gstate = PyGILState_Ensure();
int res = Py_FinalizeEx(); // <--- HANGS!!!


Debugging and reading the code revealed it hangs during joining of threads. I can reproduce the deadlock by running the following code with PyRun_SimpleString (running it right before Py_FinalizeEx):

import threading
for t in threading.enumerate():
print('get_ident: {} ; native: {}'.format(t.ident, t.native_id))
if not threading.current_thread().ident == t.ident:
t.join()


Last, I am not using PyEval_SaveThread/RestoreThread, maybe I have to, but I don't understand how to use them with GIL_Ensure/Release as I saw they are internally also taking and dropping the GIL.

Any ideas why the deadlock occurs and how to resolve this issue? Thanks!

Continue reading...