[Python] cv2.warpPerspective mean runtime increase 10x with more calls (memory leak?)

Context:

PaddleOCR ensembles detection and recognition models. Four images are being used to test with. Each image, when passed through the detector, yields 26-33 bounding boxes (118 total) which are then passed to the get_rotate_crop_image function (which calls cv2.warpPerspective) to prepare inputs to the recognition model.

Problem:

After modifying the calling code as part of an effort to support batched inference, this cv2.warpPerspective call is, on average, taking an order of magnitude longer to run.

Investigation:

In the sequential case:

• get_rotate_crop_image takes 0.012678146362304688 seconds, or 0.78%, of the total time
• the average runtime of cv2.warpPerspective is 0.00028243048477172853 seconds

In the batching case:

• get_rotate_crop_image takes 0.4442577362060547, or 23.85%, of the total time
• the average runtime of cv2.warpPerspective is 0.0037425978709075412 seconds

It’s been confirmed that the number of calls to get_rotate_crop_image is the same in both cases.

I’ve since made a change to run get_rotate_crop_image in a thread pool; this does offset the latency degradation, but the average runtime of cv2.warpPerspective remains 10x what it was in the sequential case, so performance is being left on the table.

def process_image(index_bbox):
index, bbox = index_bbox
image = ori_im[index]

return get_rotate_crop_image(image, bbox)

with ThreadPoolExecutor() as executor:
img_crop_list = list(
executor.map(
process_image,
[
(index, box)
for index, batch in enumerate(dt_boxes)
for box in batch
],
)
)


Hypothesis #1: (disproved)

Due to scaling differences caused by preprocessing image batches, the cv2.warpPerspective call might be operating on larger images and, therefore, using more memory.

However, upon inspecting the input image and points arrays, they’re similar; in fact, the image is smaller in the batched inference case:

In [1]: import numpy as np
...: from PIL import Image
...: seq_img = Image.open(“sequential/max_time_source.png”)
...: seq_points = np.load(“sequential/max_time_points.npy”)
...: batch_img = Image.open(“batch/max_time_source.png”)
...: batch_points = np.load(“batch/max_time_points.npy”)
...: print(
...: f”Sequential image shape: {seq_img.size}\n”
...: f”Batch image shape: {batch_img.size}\n”
...: f”Sequential points:\n{seq_points}\n”
...: f”Batch points:\n{batch_points}”
...: )
Sequential image shape: (1130, 842)
Batch image shape: (960, 768)
Sequential points:
[[ 979. 185.]
[1073. 188.]
[1072. 208.]
[ 978. 206.]]
Batch points:
[[832. 159.]
[911. 159.]
[911. 176.]
[832. 176.]]


To confirm that the increased runtime cannot be attributed to differences in the function’s inputs, get_rotate_crop_image was run 1000 times each with the two sets of inputs and the runtimes compared -- the differences are negligible.

If interested to repro this, the two sets of inputs have been uploaded to Google Drive.

Hypothesis #2

Memory isn’t being freed after each of these 118 calls to get_rotate_crop_image / cv2.warpPerspective. Memory allocated for the ~30 calls per PaddleOCR library call (sequential case) is comparatively less than the memory used in the batch case. In the latter case, the CPU is thrashing causing the observed slow down.

I’m not very experienced with memory profiling, but I gave memory_profiler a try and generated the charts below which confirm increased memory usage in the batching case...

I might be mistaken, but it doesn’t seem to me that ~930 MB of memory usage compared to ~700 MB would make that much of a difference, especially since the machine used to profile has 16 GB of memory and the slowdown is also observed on machines with much more memory than that (128 GB).

I’m unsure what to do at this point.

Sequential memory profile:

Batch memory profile:

Continue reading...