Architectures · intermediate · concept 88 of 311 · Updated Nov 4, 2026
Normalization Layers (BatchNorm to RMSNorm)
Layers that rescale activations so training stays stable as networks get deep. BatchNorm made deep CNNs trainable, LayerNorm made transformers possible, and RMSNorm is the leaner variant inside most current LLMs. Unglamorous, and load-bearing for everything.
Top AI contributors on this
- Geoffrey HintonUniversity of Toronto (emeritus)
- Yoshua BengioLawZero / Université de Montréal
- Yann LeCunAMI Labs / NYU
- Andrew NgDeepLearning.AI / AI Fund
- Ilya SutskeverSafe Superintelligence Inc. (SSI)
- Andrej KarpathyAnthropic
Key terms
Batch normalizationLayer normalizationRMSNormInternal covariate shiftTraining stability
Learn these first
Where you meet it in the real world
Every deep network trained since 2015; pre-norm vs post-norm is a real LLM design decision
Videos
▶ Why Does Batch Norm Work? (C2W3L06) ↗
DeepLearningAI · YouTube
▶ Group Normalization (Paper Explained) ↗
Yannic Kilcher · YouTube
Guides and articles
CS231n course notes ↗
Stanford
8.5. Batch Normalization: Dive into Deep Learning 1.0.3 documentation ↗
Dive into Deep Learning
Courses, papers, and more