Encyclopedia · 176 concepts

Math & Optimization · intermediate · concept 47 of 176

Optimizers: SGD, Momentum & Adam

Gradient descent says which direction to step; optimizers decide how big and how smart the step is. Momentum smooths the path, Adam adapts the step size per parameter, and AdamW is the default that trains nearly every modern model. Small detail, enormous practical consequence.

Key terms

SGDMomentumAdam & AdamWLearning rate scheduleWarmup

Learn these first

Where you meet it in the real world

Every training run; learning-rate schedules are often the difference between converging and diverging

Videos

Guides and articles