Math & Optimization · intermediate · concept 65 of 311 · Updated Nov 4, 2026
Optimizers: SGD, Momentum & Adam
Gradient descent says which direction to step; optimizers decide how big and how smart the step is. Momentum smooths the path, Adam adapts the step size per parameter, and AdamW is the default that trains nearly every modern model. Small detail, enormous practical consequence.
Top AI contributors on this
- Geoffrey HintonUniversity of Toronto (emeritus)
- Yoshua BengioLawZero / Université de Montréal
- Yann LeCunAMI Labs / NYU
- Andrew NgDeepLearning.AI / AI Fund
- Ilya SutskeverSafe Superintelligence Inc. (SSI)
- Andrej KarpathyAnthropic
Key terms
SGDMomentumAdam & AdamWLearning rate scheduleWarmup
Learn these first
Where you meet it in the real world
Every training run; learning-rate schedules are often the difference between converging and diverging
Videos
▶ Adam Optimization Algorithm (C2W2L08) ↗
DeepLearningAI · YouTube
Guides and articles
CS231n course notes ↗
Stanford
Dive into Deep Learning ↗
Aston Zhang and others
An overview of gradient descent optimization algorithms ↗
Sebastian Ruder
12. Optimization Algorithms: Dive into Deep Learning 1.0.3 documentation ↗
Dive into Deep Learning