[Python] How should CPU time be computed for calculations parallelized with the...

I am trying to measure the processing time, or CPU time, of a CPU-intensive computation that has been parallelized with multiprocessing. However, simply bookending the parallelization of the computation with process_time() calls and taking the difference appears to be insufficient to do this. For example, running the MRE

from time import perf_counter as pc, process_time as pt
from multiprocessing import Pool

n_workers = 2

def worker(k):
total = 0
for i in range(2*10**7):
total += i
return total

if __name__ == '__main__':

print('Serial computation')
pc_start = pc()
pt_start = pt()
results = n_workers * [0]
for k in range(n_workers):
results[k] = worker(k)
print(f' Total run time: {pc() - pc_start} seconds')
print(f' Total CPU time: {pt() - pt_start} seconds')

print('Parallel computation')
pc_start = pc()
pt_start = pt()
with Pool() as pool:
results = pool.map(worker, range(n_workers))
print(f' Total run time: {pc() - pc_start} seconds')
print(f' Total CPU time: {pt() - pt_start} seconds')


produces

Serial computation
Total run time: 1.8759662999982538 seconds
Total CPU time: 1.859375 seconds
Parallel computation
Total run time: 1.2482177000019874 seconds
Total CPU time: 0.046875 seconds


The CPU time of the second computation—the parallel one—looks highly suspect. I am running this code on a vanilla Windows 10 machine, so such a drop in CPU time relative to run time seems implausible. Is there a more accurate way to measure total CPU time when a calculation has been parallelized with multiprocessing?

Continue reading...