[Python] How to create a python module in C++ that multiprocessing does not support

I am trying and failing to reproduce and understand a problem I saw where multiprocessing failed when using a python module written in C++. My understanding was that the problem is that multiprocessing needs to pickle the function it is using. So I made my_module.cpp as follows:

#include <pybind11/pybind11.h>

int add(int input_number) {
return input_number + 10;
}

PYBIND11_MODULE(my_module, m) {
m.doc() = "A simple module implemented in C++ to add 10 to a number.";
m.def("add", &add, "Add 10 to a number");
}


After

pip install pybind11


I compiled with:

c++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) my_module.cpp -o my_module$(python3-config --extension-suffix)


I can import my_module and it works as expected.

I can test if it can be pickled with:

import my_module
import pickle

# Use the add function
print(my_module.add(5)) # Outputs: 15

# Attempt to pickle the module
try:
pickle.dumps(my_module)
except TypeError as e:
print(f"Pickling error: {e}") # Expected error


which outputs Pickling error: cannot pickle 'module' object as expected.

Now I tested multiprocessing and was surprising that it worked. I was expecting it to give a pickling error.

import my_module
from multiprocessing import Pool

# A wrapper function to call the C++ add function
def parallel_add(number):
return my_module.add(number)

if __name__ == "__main__":
numbers = [1, 2, 3, 4, 5]

try:
# Create a pool of worker processes
with Pool(processes=2) as pool:
results = pool.map(parallel_add, numbers)
print(results) # If successful, prints the results
except Exception as e:
print(f"Multiprocessing error: {e}")


How can I make a Python module in C++ with pybind11 which fails with multiprocessing because of a pickling error?

I am using Linux

Continue reading...