to give some context first I'm trying to implement a news articles similarity score. The issue I ran up to is that the model I'm using can only take 768 tokens (?) but when I limit it to that it gives me an error about the tensor size being 512. First of all, what resources can I use to learn more about this, as this is an ongoing learning and doing. Secondly, this is the function to get the embeddings (Python): def get_embeddings_bert(text: str, tokenizer, model) -> list: inputs = tokenizer( text, max_length=512, padding='max_length', truncation=True, return_tensors="pt" ) output = model(**inputs, output_hidden_states=True) document_embedding = output.hidden_states[-1] document_embedding = torch.mean(document_embedding[0], dim=1) # average document_embedding_list = document_embedding.squeeze().detach().numpy().tolist() return document_embedding_list Some months ago, I had an assignment were I needed to do something similar, it turned out great but I can't really replicate the same thing I did there, in here. In that specific case I chunked the text and then feed it to the tokenizer. Anyway, any help is appreciated! Thanks! Continue reading...
