My goal is to classify and show the types of errors in log files. I've solved the part of clustering them into buckets, but now I want to show the user the error string that is common across all the errors in each bucket. Right now I'm showing the user the first string of the bucket, but it would be nicer to show them a glob expression or a regex that matches all the strings in the bucket. For example, if I have these error strings in a bucket: "Error: File foo has 5 unsolved dependencies and 4 errors." "Error: File bar has 123 unsolved dependencies." "Error: File baz has 45 unsolved dependencies and 3 warnings." The common glob string that matches all of them would be: "Error: File * has * unsolved dependencies*." I'm working on Python, but if there's no library that can help me, any non-language specific pseudo-code is also welcome. To cluster the strings I'm using difflib. Here's the code, I'm not sure it's relevant for the question, but I wonder if I could use more of difflib's functionality to also help create this algorithm to extract common parts between strings. def cluster_strings(strings: list[str], threshold: float = 0.9) -> list[list[str]]: clusters = [] cluster_num = 0 while len(strings) > 0: string = strings.pop() clusters.append([string]) indexes_to_remove = [] for idx, string_to_match in enumerate(strings): if difflib.SequenceMatcher(None, string, string_to_match).quick_ratio() > threshold: clusters[cluster_num].append(string_to_match) indexes_to_remove.append(idx) strings = [string for idx, string in enumerate(strings) if idx not in indexes_to_remove] cluster_num += 1 return clusters Continue reading...
