I have a set of x and y data. When plotted, the data is linear except for a particular section where the data deviates from a straight line, something like a hump. My target is to develop a procedure that identifies the points in the hump (i.e. those points that deviate from the straight line). The attached image shows exactly what I want to achieve. Sample of typical plot I have tried fitting a linear trendline to the enter data, calculating the residuals and excluding the section with larger residuals; however, my approach hasn't been very successful. The trendline doesn't seem to pass through the right points. Here is my code: import numpy as np import matplotlib.pyplot as plt from sklearn.linear_model import LinearRegression # Data provided x = np.array([134, 147, 161, 175, 190, 206, 222, 237, 251, 263, 275, 291, 300, 312, 324, 337, 349, 360, 372, 382]).reshape(-1, 1) y = np.array([0.788875116, 0.692846919, 0.605305046, 0.738780558, 0.826074803, 0.871572936, 0.776701184, 0.646403726, 0.677606953, 0.615950052, 0.357934847, 0.267171728, 0.217483944, 0.155336037, 0.071882007, 0.029383778, -0.008773924, -0.050609993, -0.102372909, -0.148741651]) # Fit linear regression model = LinearRegression() model.fit(x, y) y_pred = model.predict(x) # Calculate residuals residuals = y - y_pred std_dev = np.std(residuals) # Identify hump points by adjusting alpha alpha = 2 / 3 threshold = std_dev * alpha hump_points = np.where(np.abs(residuals) > threshold)[0] # Visualize plt.figure(figsize=(12, 6)) plt.plot(x, y, 'o-', label='Data', markersize=6) plt.plot(x, y_pred, 'r--', label='Fitted Line', linewidth=2) plt.axhline(0, color='gray', linestyle='--', label='Zero Residual') plt.scatter(x[hump_points], y[hump_points], color='orange', label='Hump Point', s=100) plt.legend() plt.xlabel('X') plt.ylabel('Y') plt.title('Detecting Hump in Data') plt.grid(True) plt.show() # Print hump points hump_info = [(x[point][0], y[point]) for point in hump_points] Result of my code Continue reading...
