[Python] How to correctly design a reward function for a two-sided trading environment using...

I'm working on a high-frequency trading (HFT) reinforcement learning project using custom Gym environment and Stable Baselines3 with crypto data (BTC, 5-second bars, ~750,000 rows).

I'm facing serious issues with reward function design and environment logic in a two-sided trading setup (Long/Short/Flat) when using PPO:

• Agent often results in zero or negative total profit.
• Win rate is extremely low or 0%.
• Sharpe ratio is unrealistically negative or NaN.
• Even after implementing actions: Hold, Open Long, Close Long, Open Short, Close Short, the agent does not learn profit

What I've tried:

• Algorithms: PPO, DDPG.
• Reward: realized profit when closing a position (reward = (exit_price - entry_price) / entry_price * 100).
• Normalized data using MinMaxScaler.
• Custom two-sided trading environment (inspired by FinRL & Stable Baselines3).
• Included transaction costs and holding penalties.

Yet, the agent fails to learn meaningful strategies or profits.

My questions:

1. 
   How should I correctly design the reward function for two-sided trading?
   
   • Reward only on position closing?
   • Add floating (unrealized) PnL during holding periods?
   • Penalize overtrading, slippage, or inactivity?
2. 
   Is PPO suitable for HFT on crypto (5-sec bars)? Or should I switch to DDPG, SAC, or TD3?
   
   
3. 
   Is it wrong to normalize prices in this context? Should I keep real price scales (e.g., BTC ~90000)?
   

Example setup:


# Action Space:
# 0: Hold
# 1: Open Long
# 2: Close Long
# 3: Open Short
# 4: Close Short

# Reward (example when closing):
reward = (exit_price - entry_price) / entry_price * 100

Dataset:
BTC/USDT 5-second OHLCV.
~750,000 rows (~40 trading days).
[enter image description here](https://i.sstatic.net/oq61r4A4.png)

Continue reading...