[Python] Surprising lack of speedup in caching numpy calculations

I need to do a lot of calculations on numpy arrays, with some of the calculations being repeated. I had the idea of caching the results, but observe that

1. In most cases, the cached version is slower than just carrying out all calculations.
2. Not only is the cached version slower, line profiling also indicates that the absolute time spent on numpy operations increase, even though there are fewer of them.

I can accept the first observation by some combined magic of numpy and the python interpreter, but the second observation makes no sense to me. I also see similar behavior when operating on scipy sparse matrices.

The full application is complex, but the behavior can be reproduced by the following:

import numpy as np
from time import time

def numpy_comparison(do_cache: bool, array_size: int, num_arrays: int, num_iter: int):
# Create random arrays
arrays: dict[int, np.ndarray] = {}
for i in range(num_arrays):
arrays = np.random.rand(array_size)

if do_cache: # Set up the cache if needed - I cannot use lru_cache or similar in practice
cache: dict[tuple[int, int], np.ndarray] = {}

for _ in range(num_iter): # Loop over random pairs of array, add, store if relevant
i, j = np.random.randint(num_arrays, size=2)

if do_cache and (i, j) in cache:
a = cache[(i, j)] # a is not used further here, but would be in the real case
else:
a = arrays + arrays[j]
if do_cache:
cache[(i, j)] = a


Now running (with no multithreading)

%timeit numpy_comparison(do_cache=False, array_size=10000, num_arrays=100, num_iter=num_iter)
%timeit numpy_comparison(do_cache=True, array_size=10000, num_arrays=100, num_iter=num_iter)


gives the following results

num_iter No caching With caching
100 10.3ms 13.7ms
1000 28.8ms 62.7ms
10000 225ms 392ms
100000 2.12s 1.62s

Varying the array size and number of arrays give similar behavior. When num_iter is sufficiently high, retrieving from cache is most efficient, but in the regime relevant for my application, num_iter=1000 when the average chance of hitting a cached value is about 5%. Line profiling indicates this is not caused by working on cache, but on the addition of the arrays being slow.

Can anyone give a hint of what is going on here?

Continue reading...