Core ML Concepts · intermediate · concept 28 of 176
Data Leakage
When information from outside the training set sneaks into training, so the model aces validation and faceplants in production. Classic forms: preprocessing before the split, target information hiding in a feature, duplicate rows across splits, and peeking into the future on time series. The number-one practical reason ML projects report great numbers and then fail.
Key terms
Target leakageTrain-test contaminationTemporal look-aheadPipeline orderingBenchmark contamination
Learn these first
Where you meet it in the real world
Why deployed models underperform their offline metrics; also why LLM benchmark scores get contested
Videos
▶ Live 2020-01-20!!! Favorite ML, Data Leakage, How to Learn ML ↗
StatQuest with Josh Starmer · YouTube
Guides and articles
Courses, papers, and more