[Python] FAISS RuntimError

I want to index some documents with embedding created by ColBERTV2 in FAISS. I'm using RAGatouille framework to do this. But for some documents I'm getting following error.

EXCEPTION: Error in void faiss::Clustering::train_encoded(faiss::idx_t, const uint8_t*, const faiss::Index*, faiss::Index&, const float*) at /project/faiss/faiss/Clustering.cpp:261: Error: 'nx >= k' failed: Number of training points (51) should be at least as large as number of clusters (64)


The code i used as follows:

from ragatouille import RAGPretrainedModel
import json


output_file = "output_collection.json"

with open(output_file, "r") as f:
doc= json.load(f)


document: str = doc["text"]
id = str(doc["_id"])
index = doc["index"]

RAG = RAGPretrainedModel.from_pretrained("./colbertv2.0")

RAG.index(
collection=[document],
document_ids=[id],
document_metadatas=[
{"mongo_id": id}
],
index_name=index,
max_document_length=256,
split_documents=True,
use_faiss=True,
)


What cause this error, how can I mitigate this. This error doesn't depend on word count. Thank you

Continue reading...