Encyclopedia · 176 concepts

MLOps & Infrastructure · advanced · concept 148 of 176

Numerical Precision & Mixed-Precision Training

How many bits each number gets, and why halving them roughly doubles your throughput. Training moved from FP32 to mixed precision with BF16, and frontier runs now use FP8 for parts of the math while keeping sensitive accumulations wider. Push too far and gradients underflow or the loss diverges, so precision is a tuning decision, not just a hardware feature.

Key terms

FP32 & BF16FP8Mixed precisionLoss scalingNumerical stability

Learn these first

Where you meet it in the real world

Every large training run's cost model; the difference between a run that fits your budget and one that does not

Videos

Guides and articles

Courses, papers, and more