NLP & Language · advanced · concept 149 of 311 · Updated Nov 4, 2026

🧹Data Deduplication and Pruning

Deduplication removes repeated documents and passages from a training corpus; pruning goes further and drops low-value documents by quality or similarity. Both matter because web crawls repeat text at every scale, and Lee et al. showed models trained on deduplicated data emit memorized text about ten times less often with equal or better accuracy. Example: FineWeb runs MinHash near-duplicate removal on each Common Crawl snapshot.