I have a host with 2 cuda devices and I have a lot of text chunks to embed. As part of my project I'm using SentenceTransformer with the model BAAI/bge-small-en. I'm having difficulty making use of the 2nd device. As I understand it, the following code will instantiate a model and load it onto the first cuda device. In this implementation, model.encode can batch about 100 chunks / second: model = SentenceTransformer('BAAI/bge-small-en', trust_remote_code=True, device='cuda') However, if I try to use multi_process encoding to engage both devices, the performance drops down to about 0.5 chunks / second: pool = model.start_multi_process_pool() def encode(docs): return model.encode_multi_process(docs, pool=pool) This rate held for batch sizes 16 through 256 and for larger batch sizes, performance was even worse. Is there something else I can do to take advantage of the second cuda device, or am I doing something incorrect with the encode_multi_process method? Continue reading...
