NLP & Language · advanced · concept 150 of 311 · Updated Nov 4, 2026

📚Pretraining Data and Data Mixtures

Pretraining data is the corpus a language model learns from before any fine-tuning: web crawls, code, books, papers and forums, usually trillions of tokens. A data mixture is the share given to each source. It matters because the mixture sets what a model knows and how it writes. Example: DoReMi tuned domain weights with a small proxy model and reached the baseline's downstream accuracy 2.6 times faster on The Pile.