We are looking for ways to improve this small step in a large pipeline we are developing. The problem is: Given: multiple np.ndarray objects of integer datatype that effectively describe pixel indices and a single np.ndarray of floating type that describes a weight. All of these arrays will have the same shape/size. Problem: any time a set of indices is repeated, sum the weights for the repeated tuple (which is now unique by construction). Return: the unique indices and the summed weights. We've developed a fairly robust method, but it's quite slow. Consider: x = np.array([1, 1, 2, 2, 3, 3, 1, 1, 3, 3, 4], dtype=np.uint16) y = np.array([1, 1, 2, 2, 2, 2, 1, 1, 3, 4, 5], dtype=np.uint16) l = np.array([1, 2, 2, 2, 3, 2, 1, 1, 3, 3, 6], dtype=np.uint16) v = np.array([2, 4, 6, 8, 7, 5, 3, 1, 8, 6, 4], dtype=np.float64) indices = (x, y, l) dims = [np.amax(index) + 1 for index in indices] idx = np.ravel_multi_index(indices, dims, order='F') out, uind, cinv = np.unique(idx, return_index=True, return_inverse=True) vv = np.bincount(cinv, weights=v) out = tuple(index[uind] for index in indices) ret = (vv, *out) This works and returns the expected result: print(vv, out) array([ 6., 4., 14., 5., 7., 8., 6., 4.]) array([1, 1, 2, 3, 3, 3, 3, 4], dtype=uint16) array([1, 1, 2, 2, 2, 3, 4, 5], dtype=uint16) array([1, 2, 2, 2, 3, 3, 3, 6], dtype=uint16) This is a MWE with small arrays, but in practice, these arrays will be well in excess of a million elements. And this raises the problem that the call to np.unique with these settings is very slow. We've tried a few things: using CSR_matrix or numba, and for different reasons they weren't much better. What would be a more efficient way of performing these types of calculations? Continue reading...
