[Python] Why do two identical NumPy arrays vastly differ in computational speed?

I am working on a larger project, in the context of which I have to compute a Pandas DataFrame and convert it NumPy array before performing some matrix operations. I have found two different methods of computing this DataFrame, the latter of which is slightly faster. Assume for the moment that df, total_indices, and missing_indices are defined. The first method is:

arr1 = pandas.concat([
df,
pandas.DataFrame(dict.fromkeys(list(missing_indices),0), index=df.index)
], axis=1).T.sort_index().to_numpy()


and the second (faster) method is:

arr2 = df.T.reindex(total_indices, fill_value=0).to_numpy()


The only difference here pertains to the method by which I "complete" the DataFrame by inserting zero-valued rows for every missing row (by index) of the DataFrame df. In the first method, I concatenate df with a null DataFrame of the missing indices, whereas in the second method, I simply use .reindex() and ask Pandas to fill the missing rows with zeroes. Now keep in mind here that arr1 and arr2 are identical arrays (I have verified this with the Pandas .equals() method). Only the method of computing the arrays differ. Now when I proceed, arr2 performs vastly worse in computation. See below:

t0 = time.time()
arr1@num.transpose(arr1)
t1 = time.time()
print('Elapsed time: ', t1-t0)


takes 6.15 seconds, whereas

t0 = time.time()
arr2@num.transpose(arr2)
t1 = time.time()
print('Elapsed time: ', t1-t0)


takes a staggering 105.27 seconds.

What gives? Why is the second computation so much slower even though it is identical? If it helps to know, the arrays have a size of 338x169,543 and are sparse binary arrays (every element is a zero or one, and most elements are zero). Is there any way that I can make arr2 faster?

For a (much smaller) toy example, you could use

df = pandas.DataFrame({'1': [1,0,0,0,0,1,0,0,1,0], '3': [0,1,1,0,0,1,0,1,0,1]})
missing_indices = [0,2]
total_indices = range(4)


though I think the difference in speed will only become pronounced in a much more computationally expensive setting. Thanks for any help!

Continue reading...