[Python] Splitting multivariate time series data to avoid data leakage in PyTorch?

I have two predictors A and B with the goal to predict target (y) C using a multi-layer ANN. My inputs are 2D combinations of time series using 1 hour of A and 30 minutes of B for prediction.

Assume we are predicting C at time t = 0, then the input (x) is as follows:

x0 = [A(t=-60), A(t=-59) ... A(t=0), B(t=-30), B(t=-29) ... B(t=0)]


Then for t = 1,

x1 = [A(t=-59), A(t=-58) ... A(t=1), B(t=-29), B(t=-28) ... B(t=1)]


As you can see, inputs 0 and 1 have overlapping time history, and therefore, would have data leakage when split into a train, test and validation set like so:

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=test_split, random_state=64, shuffle=True)

x_val, x_test, y_val, y_test = train_test_split(x_test, y_test, test_size=val_split, random_state=64, shuffle=True)


One clear way to solve this issue is to make shuffle=False when splitting the dataset and then shuffling the training set in the data loader, which I do here after converting to torch tensors:

# training data loading
train_data = TensorDataset(x_train_tensor, y_train_tensor)
train_loader = DataLoader(train_data, batch_size=batch_size, shuffle=True)

# validation data loading
val_data = TensorDataset(x_val_tensor, y_val_tensor)
val_loader = DataLoader(val_data, batch_size=batch_size, shuffle=True)


The problem is that when I do not shuffle on the train/test/split, my model is awful. Suspiciously awful (e.g. correlation coefficient on the test set of -0.02). I say suspicious as I know the my predictors are statistically correlated with my target, that even if the model was not performing well I would still get better metrics than what I am seeing.

Since I do not shuffle my data during splitting, I suspect that there is important information later on in time that is in the test set that my model would need in training. I am not sure. It just seems to be performing too poorly that I think there is an issue I am not seeing.

Is there a better way to split my dataset to preserve time history and prevent major data leakage?

Edit: I also should add that I run the model through a hyperparameter tuning script, so the dataset gets to train with a variety of parameters. The results are consistently awful.

Continue reading...