[Python] Metrics that Tune Search Algorithms and Tune Trial Schedulers expects are not in the...

I'm using the new Ray API stack to implement Tune Search Algorithms and Tune Trial Schedulers, but I'm getting the following error:

ValueError: Trial returned a result which did not include the specified metric(s) `episode_return_mean` that `tune.TuneConfig()` expects. Make sure your calls to `tune.report()` include the metric, or set the TUNE_DISABLE_STRICT_METRIC_CHECKING environment variable to 1.


I'm running a code (ray==2.46.0 and python==3.12.10) very similar to the examples in the documentation:

from ray import tune
from ray.rllib.callbacks.callbacks import RLlibCallback
from ray.tune.search.optuna import OptunaSearch
from ray.rllib.algorithms.ppo import PPOConfig
from ray.tune.schedulers import ASHAScheduler, PopulationBasedTraining
from ray.rllib.utils.metrics import EPISODE_RETURN_MEAN

class MyReward(RLlibCallback):
def on_episode_end(self, *, episode, metrics_logger, **kwargs):
metrics_logger.log_value("MY_REWARD", episode.get_return(), reduce="mean", window=100)

config = (PPOConfig()
.environment("Pendulum-v1")
.framework("torch")
.training(lr=tune.uniform(0.0001, 0.01))
.callbacks(MyReward))

tuner = tune.Tuner(trainable=config.algo_class,
param_space=config,
tune_config=tune.TuneConfig(search_alg=OptunaSearch(),
scheduler=ASHAScheduler(),
metric=EPISODE_RETURN_MEAN,
mode="max",
num_samples=1))

results = tuner.fit()


From other answers (1, 2, 3, and 4), I understand that in Ray 2.x, some metrics are handled differently. However, when I run the same code without considering tune_config, the “episode_return_mean” (or EPISODE_RETURN_MEAN, or “env_runners/episode_return_mean”, or “evaluation/episode_return_mean” metric) does appear in the output dictionary, and I can also see them on tensorboard, but it still doesn’t work with OptunaSearch, ASHAScheduler, or PopulationBasedTraining.

I also tried adding MyReward callback in PPOConfig() to save a reward, and I do get this value in the output dictionary, but when I use the Tune Search Algorithms or Tune Trial Schedulers I get the same error.

What am I missing?

Thanks in advance!

Continue reading...