[Python] Does my timeseries RNN have data leakage?

I'm trying to set up an RNN to predict the generated windpower from weather data. For this I'm using a publicly available dataset from kaggle (https://www.kaggle.com/datasets/berkerisen/wind-turbine-scada-dataset/data). I'm trying to set up an RNN with tensorflow and I am very suspicous of my results and I fear that I have data leakage. My metrics are very good and the r2 value for the testing data is higher than for my training data. I think I avoided the most common pitfalls for dataleaking, i.e. splitting data before scaling and proper windowing. But as I'm quite new to machine learning I'm still unsure about this.

This is the code for just testing the RNN so no data analysis or simpler models.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# --- Import Data ---
df = pd.read_csv(r"T1.csv")

# Convert Date/Time to datetime object and split into hour/day/month/year
df['Date/Time'] = pd.to_datetime(df['Date/Time'], format='%d %m %Y %H:%M')
df['Month'] = df['Date/Time'].dt.month
df['week'] = df['Date/Time'].dt.isocalendar().week
df['day'] = df['Date/Time'].dt.day
df['Hour']=df['Date/Time'].dt.hour
df = df.drop(columns=['Theoretical_Power_Curve (KWh)','Date/Time'])

#%% --- Split, Scale and Window Data ---
from sklearn.preprocessing import MinMaxScaler

# Split data before scaling
df = df.values
train_size = int(0.8 * len(df))
test_size = len(df) - train_size
df_train, df_test = df[:train_size], df[train_size:]

# Scale only based on training data
sc = MinMaxScaler()
df_train_sc = sc.fit_transform(df_train)
df_test_sc = sc.transform(df_test)

# Split training data in to training and validation data (90/10)
val_size = int(0.1 * len(df_train_sc)) # 10% of training for validation
df_train_sc, df_val_sc = df_train_sc[:-val_size], df_train_sc[-(val_size):]

# Windowing Function
def create_sequences(data, time_step=1):
X, y = [], []
for i in range(len(data) - time_step):
X.append(data[ii + time_step), :])
y.append(data[i + time_step, 0]) # FIrst feature is target
return np.array(X), np.array(y)

time_step = 10 # adjustable

X_train, y_train = create_sequences(df_train_sc, time_step)
X_val, y_val = create_sequences(df_val_sc, time_step)
X_test, y_test = create_sequences(df_test_sc, time_step)

#%% --- Build, Compile and Train RNN ---

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Dropout

model = Sequential([
LSTM(50, return_sequences=True, input_shape=(10, 7)), # First LSTM layer
LSTM(50, return_sequences=False), # Second LSTM layer
Dense(25, activation='relu'), # Dense hidden layer
Dense(1) # Output layer
])

# Compile the model
model.compile(optimizer='adam', loss='mean_squared_error', metrics=['mae'])

history = model.fit(X_train, y_train, epochs=10, batch_size=32, validation_data=(X_val, y_val))

#%% --- Predict Windpower ---

y_pred_train = model.predict(X_train).flatten()
y_pred_test = model.predict(X_test).flatten()

num_features = 7 # Update this based on your dataset

# Create a full array for inverse transformation
y_pred_train_full = np.zeros((y_pred_train.shape[0], num_features)) # Use the same number of features as original data
y_pred_train_full[:, 0] = y_pred_train.flatten() # Insert predictions into the first column
y_pred_test_full = np.zeros((y_pred_test.shape[0], num_features)) # Use the same number of features as original data
y_pred_test_full[:, 0] = y_pred_test.flatten() # Insert predictions into the first column

# Apply inverse transformation
y_pred_train_inverse = sc.inverse_transform(y_pred_train_full)[:, 0] # Extract only the first column
y_pred_test_inverse = sc.inverse_transform(y_pred_test_full)[:, 0] # Extract only the first column

# Also invert y_train and y_test for comparison
y_train_full = np.zeros((y_train.shape[0], num_features)) # Use the same number of features
y_train_full[:, 0] = y_train.flatten()
y_train_inverse = sc.inverse_transform(y_train_full)[:, 0] # Extract only the first column
y_test_full = np.zeros((y_test.shape[0], num_features))
y_test_full[:, 0] = y_test.flatten()
y_test_inverse = sc.inverse_transform(y_test_full)[:, 0]

# calculate and print metrics
r2_train = r2_score(y_train_inverse, y_pred_train_inverse)
r2_test = r2_score(y_test_inverse, y_pred_test_inverse)
mae_train = mean_absolute_error(y_train_inverse, y_pred_train_inverse)
mae_test = mean_absolute_error(y_test_inverse, y_pred_test_inverse)

print("R² Score Training Data:", r2_train)
print("R² Score Testing Data:", r2_test)
print("MAE Score Training Data:", mae_train)
print("MAE Score Testing Data:", mae_test)


Are there any obvious mistakes? Thanks in advance

Continue reading...