Notice this post is not duplicated to any of the following relevant pieces on SO: Find The Parity Outlier Python Python: Find outliers inside a list Find outlier using z score I was given data in an experiment: import matplotlib.pyplot as plt x = [22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50] y_NaOH = [94.2, 146.2, 222.2, 276.2, 336.2, 372.2, 428.2, 542.2, 576.2, 684.2, 766.2, 848.2, 904.2, 1042.2, 1136.2] y_NaHCO3 = [232.0, 308.0, 322.0, 374.0, 436.0, 494.0, 592.0, 660.0, 704.0, 824.0, 900.0, 958.0, 1048.0, 1138.0, 1232.0] y_BaOH2 = [493.1, 533.1, 549.1, 607.1, 665.1, 731.1, 797.1, 867.1, 971.1, 1007.1, 1091.1, 1221.1, 1311.1, 1371.1, 1497.1, ] plt.plot(x, y_NaOH) plt.plot(x, y_NaHCO3) plt.plot(x, y_BaOH2) plt.show() However, I had trouble marking the outliers, here's what I have tried: import matplotlib.pyplot as plt import statistics x = [22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50] y_NaOH = [94.2, 146.2, 222.2, 276.2, 336.2, 372.2, 428.2, 542.2, 576.2, 684.2, 766.2, 848.2, 904.2, 1042.2, 1136.2] y_NaHCO3 = [232.0, 308.0, 322.0, 374.0, 436.0, 494.0, 592.0, 660.0, 704.0, 824.0, 900.0, 958.0, 1048.0, 1138.0, 1232.0] y_BaOH2 = [493.1, 533.1, 549.1, 607.1, 665.1, 731.1, 797.1, 867.1, 971.1, 1007.1, 1091.1, 1221.1, 1311.1, 1371.1, 1497.1, ] # plt.plot(x, y_NaOH) # plt.plot(x, y_NaHCO3) # plt.plot(x, y_BaOH2) # plt.show() def detect_outlier(data_1): threshold = 1 mean_1 = statistics.mean(data_1) std_1 = statistics.stdev(data_1) result_dataset = [y for y in data_1 if abs((y - mean_1)/std_1)<=threshold ] return result_dataset if __name__=="__main__": dataset = y_NaHCO3 result_dataset = detect_outlier(dataset) print(result_dataset) # [374.0, 436.0, 494.0, 592.0, 660.0, 704.0, 824.0, 900.0, 958.0] Incorrectly, this method always filter out the edge values of my data, actually I seek to remove the dots that doesn't fit the curve. Plus, I can observe the shape of the curve and mark the outliers manually, but it really costs a lot of time. I will be very grateful for your help. Expected output I want to sketch the data in line and mark the outliers as dots, for example: from matplotlib import pyplot as plt x = [22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50] y_NaOH = [94.2, 146.2, 222.2, 276.2, 336.2, 372.2, 428.2, 542.2, 576.2, 684.2, 766.2, 848.2, 904.2, 1042.2, 1136.2] y_NaHCO3 = [232.0, 308.0, 322.0, 374.0, 436.0, 494.0, 592.0, 660.0, 704.0, 824.0, 900.0, 958.0, 1048.0, 1138.0, 1232.0] y_BaOH2 = [493.1, 533.1, 549.1, 607.1, 665.1, 731.1, 797.1, 867.1, 971.1, 1007.1, 1091.1, 1221.1, 1311.1, 1371.1, 1497.1, ] o_NaOH = [542.2] o_NaHCO3 = [308.0] o_BaOH2 = [493.1] def sketch_rejected(xv, yv, y_out): nx = [] ny = [] x_out = [] for ii, dd in enumerate(yv): if dd not in y_out: nx.append(xv[ii]) ny.append(dd) else: x_out.append(xv[ii]) plt.plot(nx, ny) plt.scatter(x_out, y_out) sketch_rejected(x, y_NaOH, o_NaOH) sketch_rejected(x, y_NaHCO3, o_NaHCO3) sketch_rejected(x, y_BaOH2, o_BaOH2) plt.show() the outliers are the spiky parts of the curve which the dot doesn't fit the gradient. Instead of manually sketch each graph and identify the outliers, can I use a module to regress the data at first, then calculate the outliers. In real life, I have tons of testing result and I don't know the general equation of each. Appreciate for your help. Continue reading...
