[Python] Memory Profile and Memory Plot show different RAM usages

I am trying to reduce the memory usage of a closed source image processing (Python + OpenCV) app. When running the memory_profiler on a test script, I get a maximum memory usage of 211.4 MiB, however the python thread is using almost 3 GiB.

$ mprof run mem_Dewarper.py; mprof plot

Filename: mem_Dewarper.py

Line # Mem usage Increment Occurrences Line Contents
=============================================================
3 20.2 MiB 20.2 MiB 1 @profile
4 def run_dewarper_test():
5 20.2 MiB 0.0 MiB 1 import os, sys
6 20.2 MiB 0.0 MiB 1 import resource
7 20.2 MiB 0.0 MiB 1 from os.path import join, abspath
8
9 20.2 MiB 0.0 MiB 1 sys.path.append('./classes')
10 101.6 MiB 81.4 MiB 1 from classes.dewarper import Dewarper
11
12 101.6 MiB 0.0 MiB 1 PATH = "./"
13 101.6 MiB 0.0 MiB 1 PATH_REF = abspath( join(PATH, "v07-100.png") )
14 101.6 MiB 0.0 MiB 1 path_img = "./aOriginal.jpg"
15
16
17 ### Deskew the image
18 150.3 MiB 48.7 MiB 1 dw = Dewarper(PATH_REF, path_img)
19 164.0 MiB 13.7 MiB 1 dw.kpd_ref = dw.load_kpd("./kpd_v07.pkl")
20 211.4 MiB 47.4 MiB 1 dw.dewarp()
21
22 136.7 MiB -74.7 MiB 1 del dw




Furthermore, when I restrict the script's memory resource to 3.5 GiB, it crashes with memory errors.


# mem_Dewarper.py
@profile
def run_dewarper_test():
import os, sys
import resource
from os.path import join, abspath

os.environ['OPENBLAS_NUM_THREADS'] = '1'

# Set maximum heap size (e.g., 500 MB)
max_heap_size = 3500 * 1024 * 1024 # 3500 MB in bytes

# Set memory limit
resource.setrlimit(resource.RLIMIT_AS, (max_heap_size, max_heap_size))


sys.path.append('./classes')
from classes.dewarper import Dewarper

PATH = "./"
PATH_REF = abspath( join(PATH, "v07-100.png") )
path_img = "./aOriginal.jpg"


### Deskew the image
dw = Dewarper(PATH_REF, path_img)
dw.kpd_ref = dw.load_kpd("./kpd_v07.pkl")
dw.dewarp()

del dw


$ mprof run mem_Dewarper.py; mprof plot
mprof: Sampling memory every 0.1s
running new process
running as a Python program...
Traceback (most recent call last):
File "/home/rh/.local/share/mambaforge-pypy3/envs/test_scorer/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/rh/.local/share/mambaforge-pypy3/envs/test_scorer/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/rh/.local/share/mambaforge-pypy3/envs/test_scorer/lib/python3.9/site-packages/memory_profiler.py", line 1351, in <module>
exec_with_profiler(script_filename, prof, args.backend, script_args)
File "/home/rh/.local/share/mambaforge-pypy3/envs/test_scorer/lib/python3.9/site-packages/memory_profiler.py", line 1252, in exec_with_profiler
exec(compile(f.read(), filename, 'exec'), ns, ns)
File "mem_Dewarper.py", line 46, in <module>
run_dewarper_test()
File "/home/rh/.local/share/mambaforge-pypy3/envs/test_scorer/lib/python3.9/site-packages/memory_profiler.py", line 1188, in wrapper
val = prof(func)(*args, **kwargs)
File "/home/rh/.local/share/mambaforge-pypy3/envs/test_scorer/lib/python3.9/site-packages/memory_profiler.py", line 761, in f
return func(*args, **kwds)
File "mem_Dewarper.py", line 29, in run_dewarper_test
dw.dewarp()
File "/home/rh/.local/share/mambaforge-pypy3/envs/test_scorer/lib/python3.9/site-packages/memory_profiler.py", line 1188, in wrapper
val = prof(func)(*args, **kwargs)
File "/home/rh/.local/share/mambaforge-pypy3/envs/test_scorer/lib/python3.9/site-packages/memory_profiler.py", line 761, in f
return func(*args, **kwds)
File "/home/rh/Dev/ts/source/classes/dewarper.py", line 441, in dewarp
self.sift('img')
File "/home/rh/Dev/ts/source/classes/dewarper.py", line 268, in sift
kp, des = self.sifter.detectAndCompute(self.img, mask=None)
cv2.error: OpenCV(4.7.0) /home/conda/feedstock_root/build_artifacts/libopencv_1675729965945/work/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 192000000 bytes in function 'OutOfMemoryError'

Using last profile data.





The Dewarper class is closed source and holds a maximum of 4 images in memory:

• 4.3 MiB
• 180 KiB
• 500 KiB
• 500 KiB

Internal calls to OpenCV methods total 47.0 MiB. The class does not contain any loops.

1. What does memory_profiler actually show?
2. How can I investigate why the python thread uses over 3 GiB when the line-by-line profile shows a max usage of 211 MiB?
3. Might this have something to do with OpenCV's C++ memory heap?

Continue reading...