5 ways your ML model's score is lying to you

Description and use

Models can report inflated scores due to data leakage. By using LeakHound for detection and safesplit for leakage‑safe splits, one can ensure honest data partitions and realistic performance metrics.

Problem

Model validation scores can be inflated by data leakage, leading to overoptimistic performance estimates that fail in production.

Outcome

After adopting LeakHound and safesplit, the model’s AUC dropped from an inflated 0.971 to a realistic 0.546 on the EEG Eye State dataset, demonstrating the elimination of temporal leakage.

Workflow

• 1. Identify potential leakage sources: duplicates, near-duplicates, temporal leakage, target-encoding features, group overlap.
• 2. Use LeakHound to scan the train/test split for these leaks and report the impact on metrics.
• 3. If leaks are found, adjust the data pipeline to remove or correct them.
• 4. Use safesplit to create leakage-safe splits (group-aware, time-aware, sequence-aware).
• 5. Re-evaluate the model on the new split and verify that the score reflects true generalization.
• 6. Integrate LeakHound into CI to fail merges that introduce leaks.

Evidence

If the exact same row is in both sets, you're testing on data the model memorised.

Split it honestly (train on the earlier part of the recording, test on the later part) and it collapses to 0.546, barely better than a coin flip.

LeakHound finds the leaks. Point it at your split: pip install leakhound-ml leakhound --train train.csv --test test.csv --target label --time-col date --measure-impact

Source

safesplit stops the leaks happening in the first place. It makes leakage-safe splits: group-aware, time-aware, and sequence-aware

Source