[Python] Problem building nmslib/sklearn.feature_extraction indexes using multiple threads

I am trying to build a large in-memory lookup tool that does fuzzy matching against music data: Artist_name and Recording_name. In order to break up the search space for speed reasons, I am trying to index all of the recordings for one artist into one nmslib index, so one index per artist and then one larger index with all the artist names. The hope is to search the artist index and get one or more matches and then to search the recordings for each of the matches in hopes to implement a fuzzy search for artist/recording data.

My problem is this: When building this index, I can never get the build process to use more than about 130% of the CPU power, despite creating 8 threads to build 8 indexes in parallel. Effectively I might as well build all the indexes in one single thread, which would take too long for my needs.

import random
import string
from threading import Thread

import sklearn
from sklearn.feature_extraction.text import TfidfVectorizer
from joblib import parallel_backend
import concurrent.futures

import nmslib

MAX_THREADS = 8
MAX_INDEXES_TO_BUILD = 1000000
ITEMS_PER_INDEX = 50000
STRING_LENGTH = 100


def ngrams(string, n=3):
string = ' ' + string + ' ' # pad names for ngrams...
ngrams = zip(*[string[i:] for i in range(n)])
return [''.join(ngram) for ngram in ngrams]


def build_index():
for value, lookup_id in search_data:
lookup_strings.append(value)
lookup_ids.append(lookup_id)

lookup_ids = []
lookup_strings = []
for i in range(ITEMS_PER_INDEX):
lookup_ids.append(i)

lookup_strings.append(''.join(
random.choice(string.ascii_letters)
for i in range(STRING_LENGTH)))

vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams)
lookup_matrix = vectorizer.fit_transform(lookup_strings)

self.index = nmslib.init(method='simple_invindx',
space='negdotprod_sparse_fast',
data_type=nmslib.DataType.SPARSE_VECTOR)
self.index.addDataPointBatch(lookup_matrix, lookup_ids)
self.index.createIndex()

with parallel_backend('threading', n_jobs=MAX_THREADS):
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = []
for i in range(MAX_INDEXES_TO_BUILD):
futures.append(executor.submit(build_index))
for future in concurrent.futures.as_completed(futures):
pass


Create up to 8 threads, build 1,000,000 indexes with 50,000 items of length 100. Everything works fine, but execution is not running in parallel. (Running on Linux 5.15.0-125-generic, python 3.8.10, 128GB of Ram, 16 cores). scikit-learn==1.2.2, scipy==1.10.1, nmslib-metabrainz==2.1.3

I would expect to see a CPU utilization above 800%, but I am seeing about 130%. As you can see I added:

with parallel_backend('threading', n_jobs=MAX_THREADS):


To hopefully get more threads executing, but no luck.

Is this a limitation of nmslib (whose author is not responding to questions) or a limitation of scikit? Are there config options that I missed that might make this possible?

Continue reading...