Deep Learning

A subset of ML using neural networks with many layers (hence 'deep'). Excels at learning hierarchical representations from raw data, each layer learns increasingly abstract features.

Key terms

Hidden layersFeature hierarchyRepresentation learningEnd-to-end learning

Learn these first

← Neural Network

Where you meet it in the real world

Self-driving cars, language translation, drug discovery, artistic generation

Deep dive · 5 min

AI vs ML vs Deep Learning

These three terms get used interchangeably in headlines, and they should not be. They are nested circles. Artificial intelligence is the outer circle: any technique that makes software handle judgment-shaped tasks, including old-fashioned hand-written rules. Machine learning is the middle circle: the subset where behavior is learned from data instead of programmed. Deep learning is the inner circle: machine learning done with neural networks that stack many layers.

Why the distinction matters

A chess engine from 1997 is AI but not machine learning: humans wrote its evaluation rules. A spam filter using logistic regression is machine learning but not deep learning: it learns, but with a single simple layer. An image recognizer with fifty stacked layers is deep learning. When someone says 'the AI decided', asking which circle it lives in tells you what questions to ask: rules can be read, learned models must be tested.

Where LLMs and generative AI sit

Large language models such as ChatGPT and Claude are deep learning: neural networks with billions of parameters, trained on text. Generative AI is a usage label, not a new circle: it describes deep learning models whose output is content (text, images, audio) rather than a classification. So every LLM is deep learning, every deep learning system is machine learning, and every machine learning system is AI. None of the arrows run backward.

In one glance

AI ⊃ machine learning ⊃ deep learning: nested, not synonyms
Rules-based systems are AI but not ML
Deep = many stacked neural-network layers
LLMs and generative AI live in the innermost circle

Deep dive · 7 min

Training in Practice

The learning loop is simple; making it work reliably took the field decades of craft. Four pieces matter most in practice, and all four run live in this site's Train mode.

Initialization: the start decides the finish

Start all weights at zero and every neuron computes the same thing forever; start them carelessly and signals explode or vanish across layers. The fix is randomness with exactly the right variance: for ReLU networks, a Gaussian with standard deviation sqrt(2/n), where n is the layer's input count (He et al., 2015; the 2 exists because ReLU zeroes half the variance). The same paper's networks were the first past the human benchmark on ImageNet.

Adam: momentum plus a per-parameter memory

Plain gradient descent zigzags across ravines and crawls on plateaus. Adam keeps two running averages per parameter, the mean gradient (momentum) and the mean squared gradient (a volatility memory), and gives every one of the 13,002 dials its own adaptive step size. Its defaults from the 2015 paper, step 0.001, decay rates 0.9 and 0.999, remain the most-typed hyperparameters in machine learning.

Dropout, batches, epochs, and honest curves

Dropout randomly silences neurons during training (classically keeping each with probability 0.5) so features become redundant and robust. Data flows in mini-batches (32 examples in our demo); one full pass over the training set is an epoch. And learning curves wobble: loss jumps batch to batch, held-out accuracy sometimes dips before rising. A suspiciously smooth curve in a presentation is usually smoothed; a real one, like the one our monitor logs live, breathes.

In one glance