[Python] Share Imported Modules Across ProcessPoolExecutor spawn() Processes

I run code which uses a wrapper to iterate through a list of objects and uses a ProcessPoolExecutor to process a set of values with each object. The following code represents the process stream of my code.

Wrapper:

import importTest as iT

if __name__ == '__main__':
obj_list = list(range(3)) # list of objects
it_obj = iT.ImportTestObj()
for i in obj_list:
it_obj.outerFunc(i)


Module:

'''importTest.py'''
import time

t0 = time.time()

from concurrent.futures import ProcessPoolExecutor
from functools import partial
import xarray # an extended import time

t1 = time.time()
print(f'Import time: {t1-t0}s')

class ImportTestObj:

def __init__(self):
self.a = 1
self.b = 2
self.process_values = list(range(2))

def poolFunc(i, j, k, l):
m = i+j+k+l
return

def outerFunc(self, c):
print(c)
# do some initial processing, then pass to process pool
c+=1
with ProcessPoolExecutor(max_workers=len(self.process_values)) as p:
res = p.map(
partial(self.poolFunc, self.a, self.b, c), self.process_values
)


Running the wrapper script yields the following, where the printed times indicate when modules are loaded and how long the imports take.

Import time: 0.9825458526611328s
0
Import time: 0.9837336540222168s
Import time: 0.9939582347869873s
...
2
Import time: 0.9924976825714111s
Import time: 0.9624471664428711s


This result makes sense, since I am using a Windows machine and therefore the spawn() multiprocessing start method (multiprocessing fork() vs spawn()). The problem is that those 1-second import times amount to a significant part of the processing time of my code, approximately 20-25%. Is there a way to avoid redundant imports while still using spawn()? I cannot use fork() or threads. Perhaps there is a way to specify that only those imported modules should be shared?

Continue reading...