[Python] when using ProcessPoolExecutor i'm running out of memory

I'm trying to process huge las files and I'm running out of memory if i use ProcessPoolExecutor. If the same steps are done in for loop with a chunk of 10 times the size, I have no issue with the memory. Below is the code I've put together from internet (I'm not programmer). Looking at task manager, it feels like the code waits for all chunks to complete.

I've tested the script with a simple for loop and a chunk size of 1 million points instead of parallel execution with 10000 and it took about 30 seconds to process 16 millions points.

The file I'm trying to process with parallel ProcessPoolExecutor has about 320 millions points and with a for loop it takes about 12-14 minutes. i have files even bigger (3-4 times) than this one.

I've added a simple print in the procc function to see how many times its being accessed and on my 6 core (no HT) cpu and that function is being access a lot of times (scrolling the output) but it never get to the part to create the file at least for 1 chunk of data before i'm running out of 16+ gb of free memory out of 32gb.

Considering that a for loop can process 16 millions in 30 sec and the fact that I'm running out of memory in about 30-40 seconds and there is no file created it makes no sense to me.

with concurrent.futures.ProcessPoolExecutor() as executor:
for chunk in las.chunk_iterator(10000):
incoming.append(executor.submit(procc,chunk.array))
for z in concurrent.futures.as_completed(incoming):
try:
np_features=z.result()
np_features[:,[0,1,2]]=(np_features[:,[0,1,2]]-header.offsets)/header.scales
if not os.path.exists(dest_file):
header.point_count = 0
new_las = laspy.LasData(header)
for f in range(0,len(props)):
new_las[props[f]]=np_features[:,f]
new_las.write(dest_file)

else:
new_las = laspy.open(dest_file,"a")
added_pts = laspy.ScaleAwarePointRecord.zeros(len(z.result()),header=new_las.header)
for f in range(0,len(props)):
added_pts[props[f]]=np_features[:,f]
new_las.append_points(added_pts)
new_las.close()

except Exception as e:
print(f"Error occurred for input {futures[future]}: {e}")


Any suggestion on how to force it to write to file? I'm pretty sure that 1 chunk of 10000 points is being process way before the 30-40 seconds is up when I'll be force to reboot my laptop due to out of memory hang up

Continue reading...