Getting the following issue while trying to use add_texts to load new index records into the vector index table in Databricks. This process has worked previously without issue. The only change between now and then has been the breakdown and replacement of a "date" field into "day", "month", and "year" (each as int). There were some upstream changes to the incoming data, but they are dealt with the same way it was done before. The whole process runs the same, and I can easily see data in earlier dataframes and in the docs and texts, so I'm not sure what's going wrong. There is another data source I use this code with and there have been no issues, just on datasets that had this date change. Indexed batch 1 ERROR:langchain_community.vectorstores.databricks_vector_search:Failed to add texts to the index. Code to add is below from databricks.vector_search.client import VectorSearchClient from langchain_community.embeddings.openai import OpenAIEmbeddings # Instantiate Vector Search ## some cols remove metadata_cols = ["Postal_Code", "Day", "Month", "Year", "File_Name", "Data_Source", "Country", "City", "Timestamp", "CreatedDate_Response", "Embeddings", "Text"] vs_index = client.get_index( endpoint_name=endpoint_name, index_name=index_name_full ) # Setting the tex_column for the vector_store text_col = "" if source_table_name.startswith("Apple"): text_col = "Text_Summary" elif source_table_name.startswith("Banana"): text_col = "Text_Summary" elif source_table_name.startswith("Orange"): text_col = "Text_Summary" elif source_table_name.startswith("Pear"): text_col = "Text_Summary" vector_store = DatabricksVectorSearch( #endpoint=endpoint_name, #index_name=index_name_full, index=vs_index, embedding=embeddings, # The column name in the index that contains the text data to be embedded text_column=text_col, columns=metadata_cols ) batch_size = 100 text_splitter = RecursiveCharacterTextSplitter( chunk_size=3000, chunk_overlap=200, length_function=tiktoken_len, separators=["\n\n", "\n", " ", ""] ) def convert_datetime_to_string(metadata): for key, value in metadata.items(): if isinstance(value, (datetime, date)): metadata[key] = value.isoformat() return metadata # counter = 1 for i in tqdm(range(0, len(texts_noref), batch_size)): text_batch = texts_noref[i:i + batch_size] metadatas_batch = metadatas_noref[i:i + batch_size] text_to_index = [] metadatas_to_index = [] ids_to_index = [] for t, m in zip(text_batch, metadatas_batch): m = convert_datetime_to_string(m) if tiktoken_len(t)<=3000: text_to_index.append(t) metadatas_to_index.append(m) id = uuid.uuid4() ids_to_index.append(str(id)) # counter+=1 else: # Create chunks from the document text record_texts = text_splitter.split_text(t) # create individual metadata dicts for each chunk # this maintains the source document name for each chunk record_metadatas = [{**m} for j, text in enumerate(record_texts)] id_sub_list = [uuid.uuid4() for i in range(0, len(record_texts))] # id_sub_list = list(np.arange(counter, counter+len(record_texts))) id_list = [str(x) for x in id_sub_list] # counter = id_sub_list[-1]+1 # Append these to current batches text_to_index.extend(record_texts) metadatas_to_index.extend(record_metadatas) ids_to_index.extend(id_list) vector_store.add_texts(text_to_index,metadatas_to_index,ids=ids_to_index) # doc_batch = docs[i:i + batch_size] # vector_store.add_documents(doc_batch, ids=["1", "2", "3", "4"]) print(f"Indexed batch {i // batch_size + 1}") Continue reading...
