[Python] How can I improve mAP when optimizing YOLOv8 hyperparameters with metaheuristic...

I am working on hyperparameter optimization for YOLOv8 using a metaheuristic algorithm. Currently, I am testing the Grey Wolf Optimizer (GWO), but I plan to also experiment with other algorithms.

Here are my current settings:

1. Model: yolov8m.pt
2. Image size: 640
3. Epochs: 50
4. Iterations per run (independent trials): 100
5. Number of hyperparameters: 12 (e.g., lr0, lrf, momentum, weight_decay, warmup_epochs, warmup_momentum, augmentation parameters, etc.)
6. Dataset: DAWN dataset (tested with ~10,000 and ~4,000 images, same issue in both cases)
7. Evaluation metric: mAP50 (COCO-style)

Problem:

During optimization, the best mAP values are stuck around ~0.40, which seems quite low compared to my target (≥0.70).

The mAP does not make significant jumps during training — for example, if it starts around 0.25, it will slowly increase (0.25 → 0.26 → 0.27 …), and after 50 epochs it usually ends up around 0.40–0.43. This slow and limited growth makes me wonder if something is wrong with my setup.

I usually stop training early because training is very slow (I run experiments on Google Colab, but it becomes expensive, and on my local machine it would take days). So I interrupt when I see the mAP plateauing around 0.40, even though it might improve if I let it run longer.

What I am considering next:

• Reducing the number of hyperparameters (e.g., from 12 to 6) and running more iterations with fewer epochs, to identify good ranges for the most important hyperparameters.
• After that, reintroducing all 12 hyperparameters but initializing them with the values found in the reduced search.
• Checking whether my current search ranges for the hyperparameters might be too wide and leading to poor results.

My questions:

• What strategies can I use to improve mAP when performing hyperparameter optimization with YOLOv8 and metaheuristic algorithms?
• Are there recommended practices for selecting which hyperparameters to optimize (e.g., focusing on LR and momentum first)?
• Is early stopping at ~0.40 mAP misleading — should I let training continue longer even if progress looks flat at first?
• Could dataset size/augmentation choices be the main bottleneck here?

Hear are my some of example code (Ignore the parameters, they are sample values):

import numpy as np
from ultralytics import YOLO

# Define bounds for hyperparameters
lb = np.array([0.01, 0.1, 0.937]) # example lower bounds: lr0, lrf, momentum
ub = np.array([0.0125, 0.13, 0.98]) # example upper bounds

# Store all tried solutions
all_solutions = []

def evaluate_fitness(params):
"""
Fitness function for hyperparameter optimization.
Receives a list/array of hyperparameters, trains YOLOv8 for 1 epoch,
and returns 1 - mAP50 as fitness (lower is better).
"""
# Flatten and convert to float
if isinstance(params, np.ndarray):
params = params.flatten().tolist()
params = [float(p) for p in params]

# Extract first 3 hyperparameters (learning rate, lrf, momentum)
x0, x1, x2 = params[:3]

# Define hyperparameter dictionary for YOLO training
hyper = {
'lr0': x0,
'lrf': x1,
'momentum': x2,
'weight_decay': 0.0005,
'warmup_epochs': 3,
'warmup_momentum': 0.8,
# Additional fixed hyperparameters (augmentations etc.)
'warmup_bias_lr': 0.08,
'box': 0.04,
'cls': 0.3,
'dfl': 1.5,
'fl_gamma': 0.0,
'scale': 0.5,
'hsv_h': 0.015,
'hsv_s': 0.7,
'hsv_v': 0.4,
'degrees': 0.0,
'translate': 0.2,
'shear': 0.0,
'perspective': 0.0,
'flipud': 0.0,
'fliplr': 0.5,
'mosaic': 1.0,
'mixup': 0.15,
'copy_paste': 0.0,
}

# Load YOLO model
model = YOLO("yolov8n.pt")

# Train model for 1 epoch on a small dataset or subset
metrics = model.train(
data="data.yaml", # path to your dataset yaml
epochs=1,
imgsz=640,
**hyper
)

# Extract mAP50 metric
map50 = float(metrics["metrics"]["metrics/mAP50(B)"])

# Store solution (1 - mAP as fitness, hyperparameters)
all_solutions.append((1 - map50, hyper))

return 1 - map50

# Example: running Grey Wolf Optimizer (GWO)
# best_position = gwo(fitness=evaluate_fitness, max_iter=100, num_particles=10, dim=3, lb=lb, ub=ub)

# After GWO, you can inspect results:
# print("Best solution found:", best_position)
# print("All tried solutions:", all_solutions)

Continue reading...