[Python] How to implement a semantic similarity search system for text documents (e.g., CVs)...

I'm working on a Python application that needs to analyze and compare text documents (such as CVs) in order to provide advanced search functionality. Specifically, the goal is to allow users to:

• 
  Search for documents similar to an example file (e.g., a sample CV).
  
  
• 
  Search for relevant documents based on a free-text description (e.g., a desired professional profile).
  

I’d like to build this using Natural Language Processing (NLP) and/or Large Language Models (LLMs) to analyze and compare documents at a semantic level, rather than using plain text matching.

I initially considered using Gemini APIs to compare queries with documents, but I ran into a limitation about input token limit.

Here are my specific questions:

What’s an effective approach in Python to represent and compare text documents semantically?

Are there any recommended libraries or tools for indexing and similarity search (e.g., faiss)?

How would you manage the three types of search (similarity to an example, relevance to a free-text description, keyword-based filtering) within a single architecture?

Is it better to split documents into sections (e.g., experience, education, skills) for embedding, or process them as a whole?

What strategies are commonly used to deal with token limit issues in LLMs when processing large or many documents? (e.g., chunking, summarization, retrieval-augmented generation)

How can I ensure good performance and scalability as the number of documents increases?

I'm looking for general-purpose solutions in Python, High-level guidance, key concepts, or references to open-source projects are also very welcome.

Thanks in advance!

Continue reading...