Common Crawl
The open web corpus underlying most LLM pretraining
Common Crawl is the petabyte-scale open web archive that seeds the pretraining mix of nearly every major LLM. Its role placed it at the center of copyright litigation and the growing 'peak data' concern that high-quality tokens are being exhausted. Access to and licensing around it is now a strategic and legal chokepoint.
How it fits the stack
Common Crawl with what it depends on (above) and what it feeds (below). The figure renders as a crawlable diagram and upgrades to an interactive 3D graph as it scrolls into view.
Common Crawl in the AI stack. Common Crawl with its immediate upstream dependencies (top) and downstream dependents (bottom) in the AI value chain. Hover a node in 3D, or read the full relationships below.
Graph data (text) — 6 entities, 7 relationships
- Curated Pretraining Corpora (FineWeb, DCLM, The Stack, RedPajama) —sources→ Common Crawl
- Llama —depends on→ Common Crawl
- Llama —uses→ Common Crawl
- Meta AI (FAIR) —sources→ Common Crawl
- OpenAI —uses→ Common Crawl
- Training Data & Web Corpus —supplies→ Common Crawl
- Training Data & Web Corpus —depends on→ Common Crawl
Feeds ↓ · 8
Context — capital, rivals, policy · · 1