[Python] I can't get reproducible results with keras neural networks

As the title says. I'm currently working in python (3.11) with keras (2.12.0) and tensorflow (2.12.1, cpu-only package) libraries, and I'm facing some problems with the reproducibility of my neural networks; currently coding in jupyter notebook. I'm using a limited amount of libraries, as I am working in an enviroment. This is the scenario:

1. I import some data with pandas.
2. I train the model on a subset of the data, using the rest for testing.
3. I get different results each time I restart the kernel and fit the model.

Now, I think I'm properly setting the seeds. I read the keras documentation, the faq section on how to get reproducible results (https://keras.io/getting_started/faq/), and similar questions being asked here (How to get reproducible results in keras, Why can't I get reproducible results in Keras even though I set the random seeds?), but nothing seems to work.

I believe I did everything properly, in particular:

1. I set all kinds of seeds that are mentioned;
2. I made sure the weights are initialized the same using seeds;
3. I enabled_op_determinism, even though I'm not using the CPU, and set the PYTHONHASHSEED (i did it within the program, so that may be the problem as it's said I must do it beforehand).

Everytime I restart the kernel, the following happens:

1. I load the data (correctly)
2. Split in train-test (seeded, so it's the same everytime)
3. Create the model: I check and make sure the initial weights are the same, even the bias.
4. I fit the model: once I restart the kernel, the fit history changes slightly.

The last point is the one that confuses me. WHEN I DON'T restart the kernel, and run the model definition + estimation, I get the very same fit history each time (as I said, it's seeded). WHEN I RESTART THE KERNEL, i run everything again, and everything works up to point 3. of the last list, but then the fit history changes. How does that work? I thought it was the compiler, I am using ADAM, but I don't know? Why would that cause a problem?

I am currently stumped, I would like everything to be the same every run of the script, as I am doing simulations, and I need the results to be reproducible.

import os #Necessario per la riproducibilità
os.environ['PYTHONHASHSEED'] = '0'

import random #Random seed
random.seed(5014)

import numpy as np
np.random.seed(5014)

#Pacchetti generici
import sys
import feather #Per leggere i dataset
import time

#Reti neurali
import pandas as pd
import tensorflow as tf
tf.random.set_seed(5014)
#tf.compat.v1.set_random_seed(5014)

import keras
keras.utils.set_random_seed(5014) #This should set again all of the seeds above, I know, but I'm trying everything.

from sklearn.model_selection import train_test_split
import math
print(sys.version)


#This isn't enough to get the same results.
tf.config.experimental.enable_op_determinism()

from keras import backend as K
session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)
tf.compat.v1.keras.backend.set_session(sess)


#Loading the data
data = feather.read_dataframe("dataset1.feather") #pd.DataFrame
data.shape

#This is to get the model
def create_model(size, n_input):
#Per rendere la stima replicabile
seeds= [1000,1001,1002,1003]
print("\nSemi utilizzati per l'iniz: ")
print(seeds)

#Struttura del modello
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(size[0], input_shape=(n_input,), activation ="relu", name="Hidden1", kernel_initializer=tf.keras.initializers.RandomNormal(seed = seeds[0])))
model.add(tf.keras.layers.Dense(size[1], activation='relu', name="Hidden2", kernel_initializer=tf.keras.initializers.RandomNormal(seed = seeds[1])))
model.add(tf.keras.layers.Dense(size[2], activation='relu', name="Hidden3", kernel_initializer=tf.keras.initializers.RandomNormal(seed = seeds[2])))
model.add(tf.keras.layers.Dense(1, activation="linear", name="Output", kernel_initializer=tf.keras.initializers.RandomNormal(seed = seeds[3])))

#Metodo di stima da utilizzare
model.compile(loss='mean_squared_error', optimizer='Adam', metrics = ["mean_squared_error"])

return model


These are functions used in the code.

vars = set(data.columns)
escluse = {"output","Days"}
X = vars.difference(escluse)

#Le posizioni di Days e output
days_pos = data.shape[1]
out_pos = data.shape[1]-1

#Specifiche della rete neurale
reps = 3 #Numero di punti di partenza per ciascuna rete.
size = [200,200,200]

i = 365

print("\nCurrently: ",i)
data_to_use = data.loc[(data["Days"]<=i) & (data["Days"] >= (i-364))]

#Definire i dataset da usare per stima
train, val = train_test_split(data_to_use, test_size=0.2)
x_train = train[list(X)]
y_train = train["output"]
x_val = val[list(X)]
y_val = val["output"]

model_nn = create_model(size,data_to_use.shape[1] - 2)


Up to the lines above, everything works as it should, the model is the same everytime I restart the kernel. But then, when it comes time for

storia = model_nn.fit(x = x_train, y = y_train, batch_size = math.ceil(x_train.shape[0]/2), epochs = 1000, validation_data=(x_val,y_val), verbose =0,
callbacks = tf.keras.callbacks.EarlyStopping(monitor ="val_mean_squared_error", min_delta =0, patience =8))

storia.history["val_mean_squared_error"][0:5]


The history changes. Why? If I repeat model definition + fitting, I get the same exact results. If I restart the kernel, and run everythin again, the results change. Does anyone know how to solve it? The data is here: https://github.com/marco-pll/data-STACKoverflow.

Please, and thanks for the help. I'm about to cry.

EDIT: It was, in fact, the PYTHONHASHSEED. Just needed to put it as environment variable. Leaving the post here, as it may help someone else.

Continue reading...