I'm trying to compare the output of two tensors with different numerical precisions: one in float64 (fp64) and the other in float32 (fp32). Both tensors are computed from the same inputs and represent the same operation or result, just at different precision levels. My goal is to assess the numerical error introduced by using lower precision. Specifically, I want to compute the mean absolute difference between the two tensors: torch.mean(torch.abs(tensor_fp64 - tensor_fp32)) However, since the tensors are in different dtypes (tensor_fp64 is float64 and tensor_fp32 is float32), I'm unsure whether I should cast tensor_fp32 back to float64 before performing the subtraction and comparison. Would comparing the tensors directly as-is introduce unintended precision loss? I ran: error = torch.mean(torch.abs(tensor_fp64 - tensor_fp32)) but I realized that PyTorch will likely upcast the fp32 tensor to fp64 during the operation. I want to confirm if this is safe practice for evaluating error between precisions or if I should explicitly cast tensor_fp32 to fp64 like this: error = torch.mean(torch.abs(tensor_fp64 - tensor_fp32.to(torch.float64))) I expected that if the numerical error from using fp32 is small (i.e., negligible when compared to fp64), then the output of the mean absolute difference should also be small. I just want to make sure I'm not introducing any hidden errors or bias by comparing across dtypes. Continue reading...
