[Python] Wierd memory allocation and freeing with Numpy

So I have been tracking this weird memory leak in my Python repo for a while now. Finally I was able to isolate the problem using the code below. The numpy arrays are generating data with a similar structure to what I use - which is a nested dictionary of numpy arrays. For the longest time I thought that the nesting, or how I was reading the files with pickle, was the issue... but I just found out with this test that it's this weird Numpy issue of using a dictionary with values that are 3d arrays of dtype uint8. You can see below that memory frees when using either 2d arrays (uint8 or float) or a 2d array with uint8 -- however does not free for a 3d array with unit8. Even more odd is if I repeat the tests for a second iteration -- then all of a sudden 3d uint8 will free memory! The only thing I have found that works (finally I have a stopgap solution now) is to run the two lines of code that are toggled on with toggle=True in my code below. Something to do with allocating a new numpy array? IDK. Anyone know what is causing this? I would like a proper solution rather than the stopgap I have now. Also just curious!! thanks =)

I have also tried (to no avail): manualy triggering the garbage collector, recursively clearing up each array in the (nested) dictionary, and changing the order of allocating and freeing arrays.

you need atleast 8 gb ram to run the code below. Note that this issue is harder to detect with smaller arrays but I believe it still occurs.

import psutil
import numpy as np
import gc

# check RAM
def check_mem():
mem_dic = psutil.virtual_memory()._asdict()
for key in mem_dic:
mem_dic[key] = round(mem_dic[key]*1e-9)
return mem_dic
print(f'start {check_mem()}')

toggle = False # False does not free memory but True does free up memory

for i in range(2):
print()
print('iter', i)

data = {x:np.full((10000, 10000), 1, dtype=float) for x in range(2**3)}
print(f'make 2d float {check_mem()}')
data.clear()
print(f'free 2d float {check_mem()}')

data = {x:np.full((3, 144, 256), 1, dtype=float) for x in range(2**12)}
print(f'make 3d float {check_mem()}')
data.clear()
print(f'free 3d float {check_mem()}')

data = {x:np.full((10000, 10000), 1, dtype=np.uint8) for x in range(32)}
print(f'make 2d uint {check_mem()}')
data.clear()
print(f'free 2d uint {check_mem()}')

data = {x:np.full((3, 144, 256), 1, dtype=np.uint8) for x in range(2**15)}
print(f'make 3d uint {check_mem()}')
data.clear() # needed with toggle on otherwise does not clear mem
if toggle: # both lines below are needed with toggle on otherwise does not clear mem
data = {x:np.full((10000, 10000), 1, dtype=float) for x in range(1)}
data.clear()
gc.collect() # this doesn't affect anything, put it here to show I tried it
print(f'free 3d uint {check_mem()}')


with toggle=False I get the below results -- showing that the memory is freed up for all dictionary-arrays except for 3d uint, and only on the first iteration.

with toggle=True I get the below results -- showing that the memory is freed up for all dictionary-arrays at each iteration.

Continue reading...