[Python] Python 3.12 "can't create new thread at interpreter shutdown" Exception in...

System seeing issue on:

Windows 11
python: 3.12.1
matplotlib: 3.10.3
backend: 'tkagg'
Tkinter version: 8.6


I have a web app where I use matplotlib to generate some plots and display them in a browser and as part of the interface there is an 'interactive' button which displays the matplotlib plot windows interactively.

I am using cherrypy as the web server and the 'Interactive' button invokes an exposed method to display the plots.

Because the GUI for displaying the plots requires the call to be in the main event loop I run a separate process to display the matplotlib plots and trigger a display by putting the object that displays the plots on a queue - when the plot process gets such an object from the queue it displays the plots.

The pseudocode looks something like this:

# create cherrypy app and separate plot process
inputplotq = multiprocessing.Queue()
outputplotq = multiprocessing.Queue()

cpyapp = CPyApp(..., inputplotq, outputplotq)
plots_process = (
multiprocessing.Process(target=plot_process, args=(inputplotq,outputplotq), kwargs=keywords)
)

cherrypy.tree.mount(cpyapp)
cherrypy.engine.start()

# plotting process
def plot_process(inputplotq, outputplotq):

while not exit:
nextmsg = inputplotq.get()

if nextmsg is a plottingobj:
nextmsg.interactive_plot()
outputplotq.put("interactive windows clossed")
else
process command message

# cherry py app exposing method to generate plots
class CPyApp(..., inputplotq, outputplotq):
self.inputplotq = inputplotq
self.outputplotq = outputplotq

....

def genplots_interactive(**kwargs):
...
self.inputplotq.put(self.cachedplottingobj)
...
return self.cacheddoc

genplots_interactive.exposed = True


I initially wrote this on linux using:

python: 3.7.1
matplotlib: 3.4.0
backend: 'TkAgg'
Tkinter version: 8.6


Which seems to work fine

Then I gave it a go running on windows using:

python: 3.12.1
matplotlib: 3.10.3
backend: 'tkagg'
Tkinter version: 8.6


(I'm pretty sure the difference is due to the python version, but thought I'd mention it was on windows in case)

When I run it using this the call to

self.inputplotq.put(self.cachedplottingobj)


in the cherry py app method results in this exception:

Traceback (most recent call last):
File "C:\Users\lc922756\AppData\Local\Programs\Python\Python312\Lib\site-packages\cherrypy\_cprequest.py", line 659, in respond
self._do_respond(path_info)
File "C:\Users\lc922756\AppData\Local\Programs\Python\Python312\Lib\site-packages\cherrypy\_cprequest.py", line 718, in _do_respond
response.body = self.handler()
^^^^^^^^^^^^^^
File "C:\Users\lc922756\AppData\Local\Programs\Python\Python312\Lib\site-packages\cherrypy\lib\encoding.py", line 223, in __call__
self.body = self.oldhandler(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\lc922756\AppData\Local\Programs\Python\Python312\Lib\site-packages\cherrypy\_cpdispatch.py", line 54, in __call__
return self.callable(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\lc922756\ttrpython\ttrresultparsers\localwebserver\ttrresapp.py", line 946, in genplots_interactive
self._plots_queue.put(genplt)
File "C:\Users\lc922756\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 94, in put
self._start_thread()
File "C:\Users\lc922756\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 187, in _start_thread
self._thread.start()
File "C:\Users\lc922756\AppData\Local\Programs\Python\Python312\Lib\threading.py", line 992, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't create new thread at interpreter shutdown


I understand from my searches that this is a new protection feature added in python 3.12, but I am a bit confused as to exactly what is happening here and wondered if someone could shed some light ?

I was not aware that at the point where I call queue.put() in the cherry py app method that the interpreter was shutting down.

Any ideas on where I should look or how I might go about debugging this ? I guess it may be something to do with the way cherry py handles requests/responses.

Continue reading...