Curated Pretraining Corpora (FineWeb, DCLM, The Stack, RedPajama)
Filtered/dedup datasets that set open-model quality
High-quality filtered corpora — HuggingFace FineWeb/FineWeb-Edu, DataComp-LM, RedPajama, The Pile, and BigCode's The Stack for code — are the differentiator for open-model quality now that raw scale is commoditized. Data curation, dedup, and mixture are among the most guarded frontier trade secrets. The Stack and licensing of code data are legally sensitive.
Code corpus
The Stack (BigCode)
Trend
Quality > raw scale
How it fits the stack
Curated Pretraining Corpora (FineWeb, DCLM, The Stack, RedPajama) with what it depends on (above) and what it feeds (below). The figure renders as a crawlable diagram and upgrades to an interactive 3D graph as it scrolls into view.
Curated Pretraining Corpora (FineWeb, DCLM, The Stack, RedPajama) in the AI stack. Curated Pretraining Corpora (FineWeb, DCLM, The Stack, RedPajama) with its immediate upstream dependencies (top) and downstream dependents (bottom) in the AI value chain. Hover a node in 3D, or read the full relationships below.
Graph data (text) — 3 entities, 2 relationships
- Curated Pretraining Corpora (FineWeb, DCLM, The Stack, RedPajama) —sources→ Common Crawl
- Curated Pretraining Corpora (FineWeb, DCLM, The Stack, RedPajama) —hosts→ Hugging Face