[Python] How to add a progress bar to Python multiprocessing.Pool without slowing it down...

I'm running a long computation using multiprocessing.Pool in Python, and I wanted to add a progress bar to track it. Naturally, I tried tqdm with pool.imap, but surprisingly, it's ~7.3x slower than using pool.map() without it.

Here's a minimal example to reproduce the issue:

import numpy as np
import time
from multiprocessing import Pool, cpu_count
from functools import partial
from tqdm import tqdm


def dummy_task(step_id, size=500):
data = np.random.randn(size, 3)
dist = np.linalg.norm(data, axis=1)
return step_id, np.min(dist)


if __name__ == "__main__":
steps = list(range(500000)) # Simulate 100 frames
size = 500

print("Running normaly...")
t0 = time.time()
with Pool(processes=cpu_count()) as pool:
results = pool.map(partial(dummy_task, size=size), steps)
t1 = time.time()
pt = t1 - t0
print(f"Time taken: {pt:.3f} seconds\n")

print("Running with tqdm...")
t2 = time.time()
with Pool(processes=cpu_count()) as pool:
results = list(tqdm(pool.imap(partial(dummy_task, size=size), steps), total=len(steps)))
t3 = time.time()

pt_tqm = t3 - t2
print(f"Time taken: {pt_tqm:.3f} seconds")
print (f"Pool Process with TQDM is {pt_tqm/pt:.3f} times slower.")


There us a similar question here from six years ago. But there is no usable answer there. Here I'm looking for a way to implement a progress bar or some form of progress information without sacrificing speed. Is there any way to achieve this?

NB: I just used the imap in the tqdm example as i coudl not figure out how to use map with tqdm.

Continue reading...