🧠Overfitting & Underfitting

A model with enough parameters can score perfectly on its training data by memorizing it, the way a student can memorize past exams without understanding the subject. That failure is overfitting, and it is the central occupational hazard of machine learning. What you actually want is generalization: performance on data the model has never seen.

How you catch it

The symptom is a gap: training accuracy keeps climbing while held-out accuracy stalls or falls. This is why the train/test split from the supervised-learning lesson is sacred, and why our live browser demo reports accuracy only on 1,000 digits the network never trains on. Watch its training loss drop toward zero while held-out accuracy plateaus around 93 percent: that gap is overfitting to a 10,000-example subset, visible in real time.

The standard defenses

More data is the best medicine: the same network that plateaus near 93 percent on 10,000 digits exceeds 96 percent trained on all 60,000. Dropout randomly silences neurons during training so no neuron can lean on a partner, forcing redundant, robust features. Early stopping quits training when held-out performance stops improving. Regularization penalizes extreme parameter values. Every serious training run uses several of these at once.

The modern plot twist

Textbook theory said bigger models past a point must overfit worse. Then deep learning practice found double descent: as models grow, test error can get worse and then better again, and in specific regimes even more training data can hurt. The clean bias-variance story you may have learned is a special case, not the whole law. Honest practitioners hold both: the overfitting discipline above works, and the frontier keeps surprising the theory.