My goal is to predict the time needed to perform a task, "Task 2", given the time needed to perform an earlier task, "Task 1". I have two lists, each of which contains times required to perform a task in seconds. The two lists are linearly correlated. The two populations may or may not be normally distributed. # Each element reports time in seconds task1_s_list = [8.261, 6.768, 6.799, 8.576, 10.433, ... ] # 274 elements task2_s_list = [40.329, 28.802, 28.689, 39.818, 745.478, ...] # 274 elements I have found that the two lists are linearly correlated using the Pearson correlation coefficient. >>> correlation, p_value = pearsonr(task1_s_list , task2_s_list ) >>> print(f"Pearson correlation coefficient: {correlation}") >>> print(f"P-value: {p_value}") Pearson correlation coefficient: 0.7678790257160918 P-value: 1.5437238651528808e-54 My initial thought was to divide the values in task2_s_list by the values in task1_s_list and average the results. import numpy as np task1_s_array = np.array(task1_s_list) task2_s_array = np.array(task2_s_list) print(task2_s_array / task1_s_array) [ 4.8818545 4.25561466 4.21959112 4.64295709 71.45384837] I observed that many values are pretty similar, but other values deviate wildly, such as 71.45, above. I computed the mean and standard deviation. mean: 7.543013546 stddev: 8.317375401 If I simply multiply my independent variable by the mean, I get the formula: task2_s = 7.543 * task1_s That normally gives me a predicted task2_s that is larger than the true value, but sometimes it gives me a predicted task2_s that is much smaller than the true value. So, I know I must do something with the spread. I know that we can usually assume a dataset is normally distributed if the number of samples is high enough. My latest attempt is to randomly draw a coefficient from a normally distributed population with the mean and standard deviation I computed. coefficient = np.random.normal(loc=mean, scale=stddev) Now any given prediction might be higher or lower than the known value of task2_s. This approach sometimes produces negative values, even though task run times are always positive. I am not confident that task1_s_list and task2_s_list are actually normally distributed. How should I adjust my calcuation? Using fitter, I found that the three best distributions for task1_s_list are rel_breitwigner, alpha, and gibrat. The three best distributions for task2_s_list are norminvgauss, genhyperbolic, and cauchy. How can I better model the f(task1_s) = task2_s relationship given the lists that I have? Continue reading...
