Huggingface trainer has parameter compute_metrics. This function however receives only predictions and labels as its input: def compute_metric(pred_label): pred, label = pred_label return my_metric(pred, label) This is sufficient for computing say Acuraccy, but unsuitable for more complex metrics like NDCG, since NDCG also requires information about group_id. The typical example is information retrieval, where you benchmark set of queries (group_ids) each with different order of preferred results. Is there a way to access information about group_ids in compute_metrics? Furthermore, is there a way to access eval dataset name inside compute_metrics? Continue reading...
