[Python] For a model unsupervised without parameters is my process legit to compute the...

i have a model with no parameters, but with 1 hyperparameters for example threshold. My dataset is 20 timeseries with 20 groundtruth. What i want is to find the best hyperparameter value for it and return the score of the model

disclaimer: i cant fit() the training set and predict the whole validation set. my model takes 1 timeseries each time so in training set i just compute one by one and compute the mean f1score. Same goes for validation set. i compute the model with that particular threshold with each timeseries in the validation set and then compute the mean for the f1score

so this is my thoughts:

1. to be able to simulate how my model will work in dataset never seen before i split the dataset into training set and validation set like 19 training set and 1 validation set.
2. i use training set as testing ground and brute force all combination of my threshold from 0 to max. I found out threshold = 10 is the best in training set and it gives me f1score = 0.8 so next i need to validate the model with the validation set
3. i test it and i'm unlucky because my model has f1score=0.8 for each timeseries in training set so the mean is still 0.8 but just that single timeseries in the valdiation gives me 0.1. This score isn't correct because maybe im just unlucky. i need to perform a cross validation.
4. how to compute a cross validation? if for each new folds (new 19 training set and 1 validation set) i check the best threshold to use in the validation set, it goes against the logic of gridsearch. I need to have threshold fixed and then perform cross-validation.
5. but if i set the threshold as X what is the sense of training set? because my model doesn't fit() and in 2) i used training set to brute force the search for threshold = 10. so maybe i can just iterate for threshold 0 to max? but if that's the case, the training set is pointless and i just compute the f1score for each timeseries (20) and compute and mean f1 score. there is no point to split. for each video compute f1score and then compute the mean f1score.
6. or maybe i should compute the mean f1 score for each fold in the training set. for exmaple instead of 20timeseries, we have 3. [1,2,3]. the training set for each fold will be [1,2] [1,3] [2,3].
7. For each array i test threshold from 0 to MAX so i compute [f1_1, f1_2] and then compute the mean f1_mean1
8. then compute for the second fold [f1_1, f1_3] and then the mean f1_mean2
9. then compute for the third fold [f1_2, f1_3] and then the mean f1_mean3
10. finally i compute mean(f1_mean1, f1_mean2, f1_mean3) = f1_mean_X so the final score for the threshold = X
11. i did for each value of threshold and found out that, as we found in the beignning 10 is the best so i have f1_mean_10
12. Now instead of having that unlucky single timeseries with 0.1 in the validation set, this time i have [3] [2] [1]. for all the folds
13. i compute threshold=10 for 3, then for 2, then for 1 and then compute the mean f1 score and that's the real score of my model.

is this process legit? Or i just had to compute for each timeseries witohut splitting, and compute the cross-validation?

Continue reading...