[Python] How to Use Memory Mapping and Other Techniques to Reduce Memory Usage

I have a sample dataset of 2M vectors with 512 dimensions, each vector being a float32. I have an id field that is int64.

So, the size of raw vectors is 2M * 512 * 4 = 4GB.

The collection is memory-mapped. I have three collections with FLAT, IVF_FLAT, and HNSW indexes on the single vector field in each collection

Simply loading the collection consumes 4GB of memory, despite being memory-mapped. Querying post-loading consumes up to 5-6GB of memory, despite being memory-mapped. Releasing the collection does clear the memory - so clearly the data resides on the disk.

I have set the queryNode.mmap.mmapEnabled to true in values.yaml.

index_type, params = 'FLAT', {}
# index_type, params = 'IVF_FLAT', {'nlist': 2000}
# index_type, params = 'HNSW', {'M': 32, 'efConstruction': 200}

# Define a collection schema
eid_field = FieldSchema(name="eid", dtype=DataType.INT64, is_primary=True, description="embedding id")
embedding_field = FieldSchema(name="content_embedding", dtype=DataType.FLOAT_VECTOR, dim=512, description="content embedding")

# Set enable_dynamic_field to True if you need to use dynamic fields.
schema = CollectionSchema(fields=[eid_field, embedding_field], auto_id=False, enable_dynamic_field=True, description=f"{index_type} collection")

# Define index parameters
index_params = client.prepare_index_params()

index_params.add_index(
field_name="eid",
index_type="STL_SORT"
)

index_params.add_index(
field_name="content_embedding",
index_type=index_type,
metric_type="L2",
params=params
)

collection_name = f"benchmarking_{index_type}"
params_str = '_'.join([f"{k}_{v}" for k,v in sorted(params.items())])
if len(params_str)>0:
collection_name += f"_{params_str}"
if not client.has_collection(collection_name=collection_name):
client.create_collection(
collection_name=collection_name,
dimension=512, # The vectors we will use in this demo has 512 dimensions
schema=schema, # The collection schema we defined above
index_params=index_params,
properties={'mmap.enabled': True}
)


See: https://milvus.io/docs/mmap.md

Continue reading...