[Python] Python multiprocessing not using 100% CPU for perfectly parallel, CPU-limited problem

I am trying to parallelize the execution of a python code that solves a numerical problem. It's a relatively complicated code making heavy use of numpy. I want to solve the numerical problem for a large set of parameters. This should be easy to parallelize: just let different processes solve the problem for different regions of the parameter space.

Here is some (pseudo-)code illustrating how I attempted the parallelization over a 2D parameter space:

import numpy as np
import itertools
from multiprocessing import Pool
import MyNumericalProblemSolver

chunksize = 100
n_values_per_param = 50
n_processes = 8
p1_values = np.logspace(1,2,n_values_per_param)
p2_values = np.logspace(3,5,n_values_per_param)
param_iterator = itertools.product(p1_values,p2_values)
solver = MyNumericalProblemSolver()
def wrapper(params):
p1,p2 = params
solver.set_parameters(p1=p1,p2=p2)
solver.solve()

p = Pool(n_processes)
p.map(wrapper,param_iterator,chunksize=chunksize)


Execution of the above code typically takes several minutes. My laptop has 8 CPUs (output of multiprocessing.cpu_count()), so I expect a performance improvement of roughly a factor 8. However, what I see is an improvement of only a factor 1.5. The interesting thing is the CPU usage:

• If I don't use multiprocessing, then 1 process is using 100% CPU
• n_processes = 1: 1 process is using 100% CPU
• n_processes = 2: 2 processes, each using 100% CPU
• n_processes = 4: 4 processes, each using 50% CPU
• n_processes = 8: 8 processes, each using 25% CPU

So it looks like the maximum total usage is 200%. I don't understand why this is happening. I confirmed that when I start the program without multiprocessing in 8 different terminals, I get 8 processes each using 100%. Things I tried so far:

• change the chunksize; doesn't help, small or large chunk sizes give the same performance
• there are several similar questions mentioning that numpy (and other libraries) can mess with the core affinity (e.g. Why does multiprocessing use only a single core after I import numpy?, multiprocessing not achieving full CPU usage on dual-processor windows machine, Multiprocessing.Pool makes Numpy matrix multiplication slower). So I tried os.system(f"taskset -p 0xff {os.getpid()}") to reset the CPU affinity after importing numpy, but it doesn't change anything. If I do os.sched_getaffinity(0) before and after importing numpy, I get the same output: {0, 1, 2, 3, 4, 5, 6, 7}.

As a test to see if this is a numpy issue, I wrote this small script:

from multiprocessing import Pool
import numpy as np

matrix_dim = 100
n_processes = 8

def test_function(x):
for i in range(5000):
matrix = np.random.rand(matrix_dim,matrix_dim)
np.linalg.solve(matrix,np.ones(matrix_dim))

n_iter = 100
chunksize = 10
p = Pool(n_processes)
p.map(test_function,range(n_iter),chunksize=chunksize)


This gives 8 processes that all use 100% CPU, so it does not seem to be a numpy problem. EDIT: not always, I sometimes see the same behaviour as described above, where 8 processes use 25% CPU each. And, testing different values of n_processes between 1 and 8, no performance difference is seen.

At this point I am running out of ideas of how to tackle this issue. I am running all this in a conda virtual env using python 3.11 on Ubuntu 22.04.


Outputs requested in the comments:

from cat /proc/cpuinfo: model name: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz


Output of hwloc-ps -t when the program is running:

142024 Core:0 python
142025 Core:0 python
142026 Core:0 python
142027 Core:0 python
142028 Core:0 python
142029 Core:0 python
142030 Core:0 python
142031 Core:0 python


Output of hwloc-ls: screenshot of hwloc-ls output

Continue reading...