Math & Optimization · advanced · concept 74 of 311 · Updated Nov 4, 2026

📉Double Descent

Test error need not follow the textbook U-shape as models grow. It falls, rises to a peak near the point where the model can fit the training set (the interpolation threshold), then falls again as it keeps growing. Belkin et al. named the pattern in 2018; Nakkiran et al. showed it in CNNs, ResNets and transformers in 2019, across model size, data size and training time. It is one reason an overparameterized network can beat a smaller one.