MLOps & Infrastructure · advanced · concept 224 of 311 · Updated Nov 4, 2026
Numerical Precision & Mixed-Precision Training
How many bits each number gets, and why halving them roughly doubles your throughput. Training moved from FP32 to mixed precision with BF16, and frontier runs now use FP8 for parts of the math while keeping sensitive accumulations wider. Push too far and gradients underflow or the loss diverges, so precision is a tuning decision, not just a hardware feature.
Top AI contributors on this
- Jensen HuangNvidia
- Jeff DeanDiscovery Loop
- Clément DelangueHugging Face
- Satya NadellaMicrosoft
- Sundar PichaiAlphabet / Google
- Alexandr WangMeta
Key terms
FP32 & BF16FP8Mixed precisionLoss scalingNumerical stability
Learn these first
Where you meet it in the real world
Every large training run's cost model; the difference between a run that fits your budget and one that does not
Videos
▶ NVAITC Webinar: Automatic Mixed Precision Training in PyTorch ↗
NVIDIA Developer · YouTube
▶ NVIDIA Developer How To Series: Mixed-Precision Training ↗
NVIDIA Developer · YouTube
Guides and articles
Courses, papers, and more