Large Language Models (LLMs)

← Bag-of-Words & TF-IDF Word2Vec & Word Embeddings →

Transformers trained on internet-scale text to predict the next token, then tuned to follow instructions. That single objective, done at enough scale, produces translation, code, reasoning and conversation as side effects nobody explicitly programmed. Everything downstream on this page, prompting, RAG, fine-tuning, agents, is a technique for getting more out of that one capability.

Interactive · 3D

How LLMs Work: the 3D walkthrough →

21 stages from your keystrokes to the next token, with the full training story.

Key terms

Next-token predictionPretraining & post-trainingParametersEmergent abilitiesFrontier models

Learn these first

Where you meet it in the real world

ChatGPT and Claude, coding assistants, search summaries, translation, and most of the AI economy

Deep dive · 8 min

From Neural Networks to LLMs

Everything in this chapter scales. A large language model is built from the same unit you have been studying, the weighted sum with a nonlinearity, trained by the same loop, gradient descent with backpropagation. Three changes take you from the 13,002-parameter digit reader to a frontier LLM.

Change one: the wiring pattern

Fully connected layers treat every input position identically, which wastes the structure of language. The 2017 transformer architecture introduced attention: every token computes how relevant every other token is to it, and pulls in context accordingly. The word 'it' can look back at the noun it refers to. Attention layers stack dozens deep, exactly as our little network's layers did.

Change two: the objective

Instead of ten digit classes, the output layer scores every token in a vocabulary of tens of thousands: predict the next token. That objective is self-supervised, so the training data is simply text, at internet scale. Getting good at next-token prediction forces the model to absorb grammar, facts, style, and a surprising amount of reasoning, because they all help predict what comes next.

Change three: scale, then polish

Parameters go from thousands to hundreds of billions; training tokens to the trillions; compute to months of GPU clusters. The raw result is an internet-text simulator, so post-training reshapes it: supervised examples of helpful dialogue, then reinforcement learning from human feedback, then, most recently, reasoning training. The full pipeline, tokenizer to sampling to RLHF, is staged in this site's How LLMs Work 3D walkthrough, the companion to the network you just trained.

In one glance