MLOps & Infrastructure · advanced · concept 226 of 311 · Updated Nov 4, 2026
Gradient Checkpointing and Memory-Efficient Training
A training trick that saves GPU memory by discarding most intermediate activations during the forward pass and recomputing them when the backward pass needs them. Chen et al. (2016) showed activation memory drops from linear in the number of layers to roughly the square root, at the cost of about one extra forward pass. Hugging Face exposes it as one flag, gradient_checkpointing=True, which is part of how many fine-tuning jobs fit on one GPU.
Top AI contributors on this
- Geoffrey HintonUniversity of Toronto (emeritus)
- Yoshua BengioLawZero / Université de Montréal
- Yann LeCunAMI Labs / NYU
- Andrew NgDeepLearning.AI / AI Fund
- Ilya SutskeverSafe Superintelligence Inc. (SSI)
- Andrej KarpathyAnthropic
Key terms
Learn these first
Where you meet it in the real world
Fine-tuning LLMs on one GPU, training long-context transformers, large-batch vision training, FSDP and DeepSpeed training recipes
Videos
Guides and articles
Courses, papers, and more