Suggestions / refer links /codes are appreciated. I have a data which is having more than 1500 rows. Each row has a sentence. I am trying to find out the best method to find the most similar sentences among all. What I have tried I have tried K-mean algorithm which groups similar sentences in a cluster. But I found a drawback in which I have to pass K to create a cluster. It is hard to guess K. I tried elbo method to guess the clusters but grouping all together isn't sufficient. In this approach I am getting all the data grouped. I am looking for data which is similar above 0.90% data should be returned with ID. I tried cosine similarity in which I used TfidfVectorizer to create matrix and then passed in cosine similarity. Even this approach didn't worked properly. What I am looking for I want an approach where I can pass a threshold example 0.90 data in all rows which are similar to each other above 0.90% should be returned as a result. Data Sample ID | DESCRIPTION ----------------------------- 10 | Cancel ASN WMS Cancel ASN 11 | MAXPREDO Validation is corect 12 | Move to QC 13 | Cancel ASN WMS Cancel ASN 14 | MAXPREDO Validation is right 15 | Verify files are sent every hours for this interface from Optima 16 | MAXPREDO Validation are correct 17 | Move to QC 18 | Verify files are not sent Expected result Above data which are similar upto 0.90% should get as a result with ID ID | DESCRIPTION ----------------------------- 10 | Cancel ASN WMS Cancel ASN 13 | Cancel ASN WMS Cancel ASN 11 | MAXPREDO Validation is corect # even spelling is not correct 14 | MAXPREDO Validation is right 16 | MAXPREDO Validation are correct 12 | Move to QC 17 | Move to QC Continue reading...
