[Python] Why might a unit test produce different numbers in CI vs locally?

I've got some complex mathematical code that produces a vector. I check this vector against a known truth with assert np.nanmean((analytic_truth - computed)**2) < limit.

Locally, I see the difference comes out to 0.5868231922068594, and I set my limit to be 1, accordingly. But the CI fails, telling me the difference was actually 1.5499424623567826. (This still represents very close vectors, since they contain numbers on the order of 10^5.)

I've got all the same library versions locally and in the cloud: python 3.13.2, numpy 2.2.3, scipy 1.15.2, so it's mysterious to me. My local machine is MacOS Sequoia, and the CI is GitHub's ubuntu-latest.

What could cause a mathematical discrepancy like this? Is it really just down to the hardware?

Continue reading...