[Python] When using nanobind, How to manage GIL and objects in between different threads

I have created a small framework based on taskflow and wrapping it using nanobind. When framework finishes the graph evaluation, it can send a callback that you can register. I am trying to register a python function but the application is getting crashed.

.def("set_graph_evaluation_finish_callback", &PyScene::set_graph_evaluation_finish_callback)


void PyScene::set_graph_evaluation_finish_callback(
nb::callable cb) {
nb::gil_scoped_acquire gil;
callback = std::move(cb);
scene.setGraphEvaluationFinishCallback([this]() {
nb::gil_scoped_acquire gil;
if (this->callback)
this->callback();
std::cout << "C++ lambda reached (no GIL)\n";
});
}


When lambda function is getting called from framework, the line inside lambda

nb::gil_scoped_acquire gil;


is responsible for the crash.

Critical nanobind error: nanobind::detail::incref_check(): attempted to change the reference count of a Python object while the GIL was not held.


I believe because there are two threads, the python one and the one that's calling the lambda or I could be wrong. How do I solve this problem?

Continue reading...