5 ways your ML model's score is lying to you

Описание и применение

Модель может выдавать завышенные показатели из‑за утечки данных. Используя инструменты LeakHound и safesplit, можно обнаружить и предотвратить утечки, обеспечив честные разделения и реальную оценку производительности.

Задача

Model validation scores can be inflated by data leakage, leading to overoptimistic performance estimates that fail in production.

Результат

After adopting LeakHound and safesplit, the model’s AUC dropped from an inflated 0.971 to a realistic 0.546 on the EEG Eye State dataset, demonstrating the elimination of temporal leakage.

Рабочий процесс

• 1. Identify potential leakage sources: duplicates, near-duplicates, temporal leakage, target-encoding features, group overlap.
• 2. Use LeakHound to scan the train/test split for these leaks and report the impact on metrics.
• 3. If leaks are found, adjust the data pipeline to remove or correct them.
• 4. Use safesplit to create leakage-safe splits (group-aware, time-aware, sequence-aware).
• 5. Re-evaluate the model on the new split and verify that the score reflects true generalization.
• 6. Integrate LeakHound into CI to fail merges that introduce leaks.

Доказательства

If the exact same row is in both sets, you're testing on data the model memorised.

Split it honestly (train on the earlier part of the recording, test on the later part) and it collapses to 0.546, barely better than a coin flip.

LeakHound finds the leaks. Point it at your split: pip install leakhound-ml leakhound --train train.csv --test test.csv --target label --time-col date --measure-impact

Источник

safesplit stops the leaks happening in the first place. It makes leakage-safe splits: group-aware, time-aware, and sequence-aware

Источник