[Python] In Algotrading, How to Incrementally Calculate Features for New Live Candles,...

I'm developing a live trading bot in Python that fetches OHLCV data (e.g., 15m candles) and computes a large number of features—rolling indicators (VWAP/Volume-ADI,SMA/EMA/ATR/RSI), price action, volume, etc.—for ML-based signal generation.

To optimize for speed, I want to compute features only for new incoming candles, not recalculate for the whole dataset every cycle. However, when I do this, the features for the new candle often don't match what I get if I recalculate over the entire dataset, causing my model to give inconsistent predictions between live and batch (backtest) modes.

My Project Structure

main.py (simplified):


import pandas as pd
from ohlcv_data_util import ensure_ohlcv_updated
from signal_generator import generate_signal

# Fetches all candles from 2023
df_15m = ensure_ohlcv_updated(client, symbol, "15m", "15m_ohlcv.csv")

# Only predict for the latest closed candle (for live)
use_idx = df_15m.index[-2] # Assume last candle is still forming

features_row, signal, atr, idx = generate_signal(df_15m, force_idx=use_idx)

Place order based on signal...

signal_generator.py (simplified):



import pandas as pd
import joblib
from feature_engineering_util import run_pipeline

MODEL = joblib.load("model.pkl")
SCALER = joblib.load("scaler_selected.pkl")
FEATURES = joblib.load("selected_feature_names.pkl")

def generate_signal(df_15m, force_idx=None):
if force_idx is not None:
target_idx = force_idx
else:
target_idx = df_15m.index[-2]

# (Here's the efficiency problem Only pass a window for speed:
window = 1500
pos_end = df_15m.index.get_loc(target_idx)
df_15m_sub = df_15m.iloc[max(0, pos_end - window + 1): pos_end + 1]
features_df = run_pipeline(df_15m_sub)

if target_idx not in features_df.index:
print(f"[DEBUG] Features for {target_idx} missing")
return None, None, None, None

features_row = features_df.loc[target_idx]
X = SCALER.transform(features_row[FEATURES].values.reshape(1, -1))
proba = MODEL.predict_proba(X)[0]
cls = MODEL.classes_.tolist()
p_long = proba[cls.index(1)]
p_short = proba[cls.index(-1)]
signal = 1 if p_long > p_short and p_long > 0.3 else -1 if p_short > p_long and p_short > 0.3 else 0
atr = features_row.get("atr", 0)
return features_row, signal, atr, target_idx

feature_engineering_util.py (key idea):


import pandas as pd
import ta

def run_pipeline(df):
# Adds many rolling/stat features
df['ema_200'] = df['close'].ewm(span=200, adjust=False).mean()
df['rsi_14'] = pd.Series(...) # Rolling RSI
df = ta.add_all_ta_features(df)
# ... other features ...
df = df.dropna()
return df

The Problem


When running batch mode over the whole dataset (for backtests), my features and model predictions are as expected.

When running live mode with just the most recent window (1500 candles for speed), my features for the latest candle differ from batch mode—especially for cumulative features like VWAP, Volume-ADI, and rolling features like EMA etc.

This causes my model to give inconsistent signals live vs. backtest.

Example:

Full batch


features_full = run_pipeline(df_15m)
signal_full = model.predict(scaler.transform(features_full.iloc[-1][FEATURES].values.reshape(1, -1)))

"Live" incremental


window = 1500
features_recent = run_pipeline(df_15m.iloc[-window:])
signal_recent = model.predict(scaler.transform(features_recent.iloc[-1][FEATURES].values.reshape(1, -1)))

Often: signal_full != signal_recent # or, features_full.iloc[-1] != features_recent.iloc[-1]


What I've Tried Increasing the window (helps, but still not perfect for very long features or chained indicators).

Attempting to append only the latest new features to the main DataFrame, but state is lost for some rolling stats.

Reading docs for pandas/TA-Lib/ta/pandas-ta but not seeing a built-in pattern for this.

What I Need: How do you handle incremental (live) feature calculation for new candles, while ensuring results exactly match a batch (full history) calculation—especially for features with rolling/EMA/state?

• 
  Are there patterns or libraries for maintaining rolling state between batches?
  
  
• 
  Is there a “pro” way to cache the state from the last feature calculation to use as the starting point for the next?
  

My requirements:

• Use pandas or vectorized libraries if possible for speed.
• Deterministic: model signals must match batch/backtest and live at all times.

Any sample code, advice, or pointers to libraries/tools that handle this robustly for trading/ML is hugely appreciated!

Continue reading...