[Python] How to use subset of covariates for future forecasting with RNN (LSTM) in Darts?

I have built a time series forecasting model using RNN (LSTM) with the Darts library to predict the Open, High, Low, and Close prices of a financial instrument. My training covariates include both time-related features and derived features from the prices. However, for future forecasting, I can only use the time-related features, since the derived features cannot be accurately forecasted.

The Problem:


When attempting to create future covariates using only DOW, DOM, and WOY, I encounter the following error:

ValueError: `X has 3 features, but MinMaxScaler is expecting 8 features as input.`
​

Relevant Code Snippet:


import pandas as pd
from darts import TimeSeries
from darts.models import RNNModel
from darts.dataprocessing.transformers import Scaler

# Load and prepare the data
header = ["Date", "Time", "Open", "High", "Low", "Close", "Volume"]
df = pd.read_csv("EURUSD0414.csv", names=header)
df['DateTime'] = pd.to_datetime(df['Date'] + ' ' + df['Time'])
df.drop(columns=['Date', 'Time'], inplace=True)
df.set_index('DateTime', inplace=True)
df = df.asfreq("D")
df = df[df.index >= '2010-01-01']
df['DOW'] = df.index.dayofweek + 1
df['DOM'] = df.index.day
df['WOY'] = df.index.isocalendar().week

# Add RSI to the DataFrame
def calculate_rsi(data, column='Close', period=14):
delta = data[column].diff(1)
gain = delta.where(delta > 0, 0)
loss = -delta.where(delta < 0, 0)
avg_gain = gain.rolling(window=period, min_periods=1).mean()
avg_loss = loss.rolling(window=period, min_periods=1).mean()
rs = avg_gain / avg_loss
rsi = 100 - (100 / (1 + rs))
return rsi

df['RSI'] = calculate_rsi(df, period=14)

# Add EMA columns
def add_ema(df, col, fast=12, slow=26):
df[f"{col}_EMA_fst"] = df[col].ewm(span=fast, adjust=False).mean()
df[f"{col}_EMA_sl"] = df[col].ewm(span=slow, adjust=False).mean()
df[f"{col}_EMA_diff"] = df[f"{col}_EMA_fst"] - df[f"{col}_EMA_sl"]
return df

fast_ema, slow_ema = 12, 26
for col in ['Close', 'High', 'Low']:
add_ema(df, col=col, fast=12, slow=26)

# Declare covariates
covar_columns = ["Volume", "DOW", "DOM", "WOY", "RSI", "Close_EMA_fst", "Close_EMA_sl", "Close_EMA_diff"]

# Scale covariates
scaler_covariates = Scaler()
covariates_df = df[covar_columns]
scaled_covariates = scaler_covariates.fit_transform(covariates_df)

# Declare target columns and scale them
target_cols = ["Open", "High", "Low", "Close"]
series = TimeSeries.from_dataframe(df, value_cols=target_cols)
scaler_y = Scaler()
y_transformed = scaler_y.fit_transform(series)

# Initialize LSTM model
model = RNNModel(model="LSTM", hidden_dim=128, n_rnn_layers=1, dropout=0.1, n_epochs=300,
optimizer_kwargs={"lr": 0.007}, random_state=1502,
training_length=101, input_chunk_length=100,
pl_trainer_kwargs={"accelerator": "gpu", "devices": [0]})

# Fit the model with transformed target data and scaled covariates
model.fit(y_transformed, future_covariates=scaled_covariates)

# Create future covariates (only using time-related features)
future_dates = pd.date_range(start=df.index.max() + pd.Timedelta(days=1), periods=1, freq="D")
future_df = pd.DataFrame({'DateTime': future_dates, 'DOW': future_dates.dayofweek + 1,
'DOM': future_dates.day, 'WOY': future_dates.isocalendar().week})
future_df.set_index('DateTime', inplace=True)

# Only scale future time-related covariates
future_covariates = scaler_covariates.transform(future_df)
future_covariates = future_covariates.astype("float32")

# Forecast using the time-related features
predicted_series = model.predict(n=1, future_covariates=future_covariates)
print(predicted_series)

Questions:

1. 
   Is it possible to train the LSTM model with many covariates and then use only a subset of these covariates for future forecasting?
   
   
2. 
   If yes, how can I implement this without running into errors related to scaling the future covariates?
   
   
3. 
   If no, how can I include derived features (like Volume, RSI, EMA_fst, EMA_sl, EMA_diff) in my future covariates given that these features are derived from values being forecasted?
   

Continue reading...