[Python] Performance and Memory duplication when using multiprocessing with a multiple...

I am having difficulties understanding logic of optimizing multip-rocessing in Python 3.11.

Context:


I am running on Ubuntu 22.04, x86 12 cores / 32 threads, 128 GB memory

Concerns:


(Please refer to code and result below).
Both multiprocessing function using a local df (using map+partial or starmap) take a lot more time than multiprocessing using the global df.

It seemed ok to me but ...

... by activating the sleep(10) in both fun, I noticed that every spawned process takes around 1.2 GB of memory (using system monitor in Ubuntu), so I guess the df must have been duplicated even when declared global.

My question (eventually )

• Why multi-processings using local df take so much time compared to the one using a global df if memory is copied in each process (whether it is local or global) ?
• Or maybe system monitor in Ubuntu report the memory accessible to a process and not necessarily its intrinsic footprint ? (shared memory would then appear duplicated in several processes and the sum of all these processes could be greater than my system memory) ?

Thank you for your enlightenment

Code


from time import sleep
from multiprocessing import Pool
from functools import partial
from time import time

import numpy as np
import pandas as pd


def _fun_local(df, imax: int):
# sleep(10)
df_temp = df[:imax]
return df_temp


def _fun_global(imax: int):
global mydf
# sleep(10)
df_temp = mydf[:imax]
return df_temp


def dummy():
# Create a 1 GB df
global mydf
n_rows = 13_421_773
n_cols = 10
data = np.random.rand(n_rows, n_cols)
mydf = pd.DataFrame(data, columns=[f'col_{i}' for i in range(n_cols)])
# check memory footprint
print('mydf', mydf.memory_usage(deep=True).sum() / 1024 / 1024)

imaxs = range(1, 33)

# With local df, call function with partial
start = time()
with Pool() as pool:
fun = partial(_fun_local, mydf)
results_partial = pool.map(fun, imaxs)
print(f'local-partial took: {time()-start}')

# With local df, call function with starmap
start = time()
with Pool() as pool:
results_starmap = pool.starmap(_fun_local, [(mydf, imax) for imax in imaxs])
print(f'local-starmap took: {time()-start}')

# With global mydf
start = time()
with Pool() as pool:
results_global = pool.map(_fun_global, imaxs)
print(f'global took: {time()-start}')

return results_local, results_global

if __name__ == '__main__':
results = dummy()

Result:


mydf (MB): 1024.0001411437988
local-partial took: 89.05407881736755
local-starmap took: 88.06274890899658
global took: 0.09803605079650879

Continue reading...