Architectures · advanced · concept 108 of 311 · Updated Nov 4, 2026

✂️Pruning and Sparsity

Removing weights, neurons, heads or layers a trained network does not need, so it runs with less memory and compute. LeCun's Optimal Brain Damage set the idea in 1989. Unstructured pruning zeroes single weights; structured pruning removes blocks hardware can skip. SparseGPT prunes OPT-175B to 50% sparsity with no retraining; Wanda ranks weights by magnitude times activation. NVIDIA's 2:4 pattern gets a hardware speedup on tensor cores.