Gradient Descent

The fundamental optimization algorithm for training neural networks. It iteratively adjusts model parameters in the direction that reduces the loss function, like walking downhill to find the valley. Convex problems have a single global minimum, which is why classical optimization loved them; deep networks are wildly non-convex and work anyway.

Interactive · 3D

Watch gradient descent race on a real terrain →

Two balls, plain SGD vs momentum, on paths computed by actually running the algorithms.

Key terms

Learning rateStochastic gradient descent (SGD)MinibatchConvergenceConvexity

Deep dive · 7 min

How Machines Learn

Strip away every acronym and modern AI is one loop, run at absurd scale. Guess: the model takes an input and produces an output using its current internal numbers (its parameters). Measure: a loss function turns the gap between the guess and the truth into a single number. Nudge: every parameter is adjusted a tiny amount in the direction that shrinks that number. Repeat: millions of times, over thousands or trillions of examples.

The downhill picture

Imagine the error as a landscape where every location is one possible setting of all the model's parameters, and altitude is how wrong the model is there. Learning is walking downhill in fog: you cannot see the valley, only the slope under your feet, so you step against the slope, over and over. This is gradient descent, the idea powering everything from a 13,002-parameter digit reader to trillion-parameter language models. The mathematical trick that makes it affordable, computing the slope for every parameter at roughly the cost of two extra guesses, is called backpropagation, and it gets its own lesson.

Data is the teacher

The loop has no other source of knowledge than the examples it sees. Feed it biased examples and it learns the bias, faithfully. Feed it too few examples and it memorizes instead of generalizing. This is why data quality, coverage, and honest held-out testing (evaluating on examples the model never trained on) matter more than any single algorithm choice, and why serious teams spend most of their time on data, not model code.

Watch it happen, for real

This is not an abstraction you have to take on faith. The 3D neural network explainer on this site has a Train mode that runs this exact loop live in your browser on 10,000 real handwritten digits: you watch the guesses start random, the error fall, and accuracy on unseen digits climb past ninety percent in about a minute.

In one glance