I've been having many issues trying to run some code with multiprocessing tools. I've found many examples of using different tools for the task, but most of them were too simplistic, thus I am not sure if I used them incorrectly or was it wrong to use them at all. I have a large code that runs on several large datasets. To save time, I'd like to run the datasets in parallel. In general, my case is as follows: funA(var1, var2, var3): # Some Code # funB(varB): # Some Code # funA(x1, x2, x3) funA(y1, y2, y3) # Some Code # # Some Code # funB(z1) funB(z2) # Some Code # I'd like to run instances of funB in parallel and when the code reaches funA, I'd like its instances to parallelize as well. So far I've tried several solutions: Attempt 1: from multiprocessing import Pool funA(var1, var2, var3): # Some Code # funB(varB): # Some Code # pool1 = Pool() pool1.apply_async(funA, [x1, x2, x3]) pool1.apply_async(funA, [y1, y2, y3]) # Some Code # # Some Code # pool0 = Pool() pool0.map(funB, [z1, z2]) # Some Code # This gave me some windows error. Attempt 2: from pathos.multiprocessing import ProcessingPool, ThreadingPool amap = ProcessingPool().amap tmap = ThreadingPool().map funA(var1, var2, var3): # Some Code # funB(varB): # Some Code # tmap(funA, [x1, x2, x3]) tmap(funA, [y1, y2, y3]) # Some Code # # Some Code # amap(funB, [z1, z2]) # Some Code # This also gave me some windows error. Attempt 3: from concurrent.futures import ThreadPoolExecutor Pool1 = ThreadPoolExecutor(max_workers=2) Pool2 = ThreadPoolExecutor(max_workers=2) funA(var1, var2, var3): # Some Code # funB(varB): # Some Code # Pool2.submit(funA, x1, x2, x3) Pool2.submit(funA, y1, y2, y3) Pool2.shutdown(wait=True) # Some Code # # Some Code # Pool1.submit(funB, [z1, z2]) Pool1.shutdown(wait=True) # Some Code # This does not crash, but it doesn't execute the functions either. It skips right away to the code that follows the parallel functions. I've read that using shutdown(wait=True) should prevent that, but even after adding it, the result is the same. It doesn't seem to execute the functions. I would be very grateful if someone could explain to me how to do this properly. I am using a Windows 11 device and Anaconda with Python 3.11. Thank you all in advance. Continue reading...
