[Python] Accesing only one row/element of a df/array stored in Azure Blob from an Azure...

I have a pretty big df/np array stored in Azure blob storage and I am wondering if I can access just certain rows of it without loading the whole thing into memory. This df/array is a cosine similarity matrix of 8GB and I want to query just one of its rows ( I know the index) from an Azure Function.

On a local environment I have been able to succesfully do this but so far I have been unable to do so with files stored in Azure as every attempt ends up loading the whole blob to memory somewhere (and that of course means that the Azure function takes forever to get a result).

Is this even possible? Any tips/ideas in case it is?

Both this:

with h5py.File('pathtofile', 'r') as f:
product_index = 9
product_similarities = f['cosine_similarity'][product_index, :]


And this:

cosine_sim_matrix = np.load('pathtofile', mmap_mode='r')
product_index = 9
product_similarities = cosine_sim_matrix[product_index, :]


Seem to work ok locally (actually on Google colab so not even a real local PC) but no chance with Azure stored file.

Continue reading...