I am facing a problem in maintaining the reproducibility in the ML project. I believe the core snippet of my issue is clf = Clf(random_state=cfg.seed) # instantiate the K-fold cross-validation object cv = StratifiedKFold( n_splits=cfg.train.n_folds, shuffle=False,#cfg.train.is_shuffle, random_state=cfg.seed ) # instantiate the grid search object grid_search = TSCGridSearchCV( estimator=clf, param_grid=dict(cfg.train.grid_params), cv=cv, n_jobs=cfg.train.n_jobs, scoring=cfg.train.scoring, verbose=4-(logger.level(cfg.log.cli.level).no//10), ) Where Clf is CNNClassifier. Although it is ridiculously time-consuming, I am managing to reproduce the same results across the runs when cfg.train.n_jobs=1. To set the seeds, I am using the following function: def _set_seed(seed: int): """Sets all random seeds for reproducibility.""" # Set seed random.seed(seed) # Deterministic TF ops tf.config.experimental.enable_op_determinism() For instance, the learning curve of the best fitted model for three independent runs when n_jobs=1 is: However, when n_jobs>=2 , the training phase performs differently across runs, even though the parameter set is kept the same (previous plot with two more runs where n_jobos=4): Making runs reproducible is one of the project requirements, I need that. How to ensure reproducibility? Continue reading...
