Topic

Attention

Attention is the mechanism that lets a language model weigh every other word when it reads one word. It is why a model can tell which noun a pronoun refers to across a long sentence. Attention sits at the center of the transformer, the architecture behind today's large language models, and it is easiest to understand by watching it work.

1 guide on Attention.

Reference Guide · 18 min read

How LLMs Work

Every stage a prompt passes through, modeled as an explorable 3D machine. 21 stages, a 21-step guided journey tracing one token from "The cat sat on the" to "mat", real parameter counts from GPT-3 to DeepSeek-V3, and the training story, pretraining, RLHF, and the reasoning-RL breakthrough.

LLM Transformers Attention Mixture of Experts Reasoning Models 3D Interactive AI Explainer

Keep learning Attention

Attention for Neural Networks, Clearly Explained!!! video · StatQuest with Josh Starmer
Attention in transformers, step-by-step | Deep Learning Chapter 6 video · 3Blue1Brown
Attention is All You Need · Yannic Kilcher (Paper Walkthrough) video · Yannic Kilcher
Deep dive - Better Attention layers for Transformer models video · Julien Simon
LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLU video · Umar Jamil
MedAI #54: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness | Tri Dao video · Stanford MedAI
Mistral / Mixtral Explained: Sliding Window Attention, Sparse Mixture of Experts, Rolling Buffer video · Umar Jamil
3Blue1Brown · But what is a GPT? + Attention, step by step guide · 3blue1brown.com

From the AI Roadmap. Every link was checked.

3D Interactive AI Explainer LLM Mixture of Experts Reasoning Models Transformers

All Guides

Attention

How LLMs Work

Keep learning Attention

Related topics