[Python] Get an array containing differences between two NumPy arrays with some members the...

I have two very large NumPy arrays containing coordinates [[x1,y1,z1],[x2,y2,z2]...] (~10^9 elements)

The arrays are of different sizes, and there will be overlap between the coordinates. So [x1,y1,z1] may be in array1, but not in array2.

I would like to quickly get all of the coordinates that are in one, and only one array.

For example,

import numpy as np
array1 = np.array([[1,2,3],[2,3,4],[3,4,5],[4,5,6]])
array2 = np.array([[2,3,4],[3,4,5],[4,5,6],[5,6,7]])

array_diff = some_function_to_get_diff(array1,array2)


would get:

array_diff: np.array([[1,2,3],[5,6,7]])


I can use setdiff1d for one dimension, but it isn't clear to me how to do this for a 2d array. The files are quite large – 10GB or so, so being able to do this quickly/chunked in parallel would be a big plus.

Update

def some_faster_function_to_get_diff(array1, array2):
unique1 = np.unique(array1, axis=0) # Get the unique coordinates in array1
unique2 = np.unique(array2, axis=0) # Get the unique coordinates in array2

set1 = set(map(tuple, unique1))
set2 = set(map(tuple, unique2))

array1_only = np.array(list(set1 - set2))
array2_only = np.array(list(set2 - set1))

if array1_only.size == 0:
return array2_only
if array2_only.size == 0:
return array1_only

return np.vstack((array1_only, array2_only))


def symmetric_difference(arr1, arr2):

dtype = [('f{}'.format(i), arr1.dtype) for i in range(arr1.shape[1])]
struct_arr1 = np.ascontiguousarray(arr1).view(dtype)
struct_arr2 = np.ascontiguousarray(arr2).view(dtype)
unique_to_arr1 = np.setdiff1d(struct_arr1, struct_arr2, assume_unique=True)
unique_to_arr2 = np.setdiff1d(struct_arr2, struct_arr1,assume_unique=True)
result = np.concatenate([unique_to_arr1, unique_to_arr2]).view(arr1.dtype).reshape(-1, arr1.shape[1])

return result


from sys import getsizeof
for i in range(2,9):

no_ele=3*10**i
arr1 = np.random.rand(no_ele, 3) * 100
arr2 = np.random.rand(no_ele, 3) * 100

print(f"i={i}: memory {getsizeof(arr2)/1024**3:2f}GB")
arr2[50:60] = arr1[50:60]
t0=time.time()
diff1=symmetric_difference(arr1,arr2)
t1=time.time()
print(f"{no_ele}: contiguous {t1-t0}")
t0=time.time()
diff2=some_faster_function_to_get_diff(arr1,arr2)
t1=time.time()
print(f"{no_ele}: sets {t1-t0}")

Gives:
2 memory 0.000007GB
300: contiguous 0.0004420280456542969
300: sets 0.0007300376892089844
3 memory 0.000067GB
3000: contiguous 0.0038480758666992188
3000: sets 0.0068819522857666016
4 memory 0.000671GB
30000: contiguous 0.052056074142456055
30000: sets 0.07555103302001953
5 memory 0.006706GB
300000: contiguous 0.6442217826843262
300000: sets 0.9859399795532227
6 memory 0.067055GB
3000000: contiguous 9.833515882492065
3000000: sets 12.196370840072632
7 memory 0.670552GB
30000000: contiguous 142.16755604743958
30000000: sets 138.87937593460083


@Andrés code works, and is much quicker than using lists, converting to contiguous array also produces a similar run time.

The problem now is that the arrays are too big to fit into memory - so it seems I need to figure out a way of batching/chunking the data...

Continue reading...