[Python] Does dtype (float16 vs. bfloat16) affect LLM prediction accuracy and evaluation...

Question: Does using different dtypes for predictions (during inference) have a meaningful impact on the evaluation metrics and prediction accuracy of an LLM? Is the impact significantly different for float16 and bfloat16 (also float16 and float32)?

I'm aware that dtype influences latency and memory usage. However, it's unclear how this translates to a practical and measurable difference in evaluation performance.

Use case: I am benchmarking VLMs on some datasets on several VM instances, calculating CER, WER, F1 scores. I noticed that on A100 the models (from Gemma family) are loaded with torch_dtype = torch.bfloat16, while on V100 torch.float16 is used. Now I am unsure if I can legitimately compare the evaluation scores from the bfloat16 run on the A100 to the float16 run on the V100?

Thank you in advance!

Continue reading...