[Python] Why is the upload of files to GCP Vertex AI RAG corpora so slow?

I am experimenting with RAG on GCP/Vertex AI, and tried to create some simple example.

Here's what I came up with, creating small dummy files locally and then uploading them one by one to a newly-minted RAG corpus:

import vertexai
from vertexai import rag
from tqdm import tqdm
from pathlib import Path
import lorem

PROJECT_ID = "my-project-id" # change this as appropriate
LOCATION = "us-central1"
CORPUS_DISPLAY_NAME = f"dummy_corpus"
TEMP_FILES_DIR = Path("temp_rag_files")

def create_files(num_files=5):
"""Creates a specified number of dummy text files with lorem ipsum content."""
TEMP_FILES_DIR.mkdir(exist_ok=True)
created_file_paths = []
for i in range(num_files):
file_path = TEMP_FILES_DIR / f"dummy_file_{i+1}.txt"
content = f"Dummy file {i+1} for RAG example.\n{lorem.paragraph()}"
file_path.write_text(content, encoding='utf-8')
created_file_paths.append(file_path)
print(f"Created dummy file: {file_path}")
return created_file_paths

def main():

vertexai.init(project=PROJECT_ID, location=LOCATION)

print("Creating dummy files...")
dummy_file_paths = create_files(num_files=5)

print(f"Creating RAG Corpus '{CORPUS_DISPLAY_NAME}'...")
corpus = rag.create_corpus(
display_name=CORPUS_DISPLAY_NAME,
description="Corpus with lorem ipsum files.",
)
corpus_name = corpus.name
print(f"Successfully created RAG Corpus: {corpus_name}")

print(f"Uploading {len(dummy_file_paths)} files to '{corpus_name}'...")
uploaded_rag_files_info = []

for file_path in tqdm(dummy_file_paths):
display_name = file_path.stem
rag_file = rag.upload_file(
corpus_name=corpus_name,
path=str(file_path),
display_name=display_name,
description=f"Dummy lorem ipsum file: {display_name}",
)
uploaded_rag_files_info.append({"name": rag_file.name, "display_name": rag_file.display_name})
print(f" Successfully uploaded: {rag_file.name}")

if __name__ == "__main__":
main()



When running the code though, each file upload iteration is quite slow (10 seconds per file), thus making the upload of a reasonably-sized dataset unfeasible.

Am I doing anything wrong?

In case this could be due to latency, I also tried europe-west3 as location (closer to me), but that replacement fails with

RuntimeError: ('Failed in indexing the RagFile due to: ', {'code': 400, 'message': 'Request resource location europe-west3 does not match service location us-central1.', 'status': 'FAILED_PRECONDITION'})
​

I may resort to other ways to upload files to the corpus, but I thought this could be a reasonable approach.

Continue reading...