MLOps & Infrastructure · intermediate · concept 217 of 311 · Updated Nov 4, 2026
A/B Testing for ML
An A/B test sends real users to the old and new versions at random and measures which does better on a metric that matters. It is the only reliable way to learn whether a model that scores higher offline actually helps people. Common errors include stopping as soon as the result looks good and testing too many things at once.
Key terms
Control and treatmentStatistical significancePeekingOnline versus offline metricsGuardrail metrics
Learn these first
Where you meet it in the real world
Rolling out ranking and recommendation models, comparing prompts and LLM versions
Guides and articles
This unlocks