Machine Learning / Editorial solution guide
Investigate suspicious validation results and leakage
Trace what information each fitted step and selection decision could see before trusting a surprisingly strong score.
Read-only knowledge base · No public comments
Recognize the symptoms
Validation appears nearly perfect, results fall sharply on new entities or later dates, or a simple model performs implausibly well. These observations justify investigation but do not themselves prove leakage. Preserve the exact procedure, split identities, metric, and data version before changing anything.
Understand the likely causes
- Possible causes include future-only features, target-derived fields, related entities across splits, global preprocessing or selection, incorrect target joins, repeated final-test tuning, or an easy synthetic task mistaken for external evidence.
- A faithful evaluation can also differ from deployment because the populations or decision policies differ. Diagnose the information boundary rather than assuming every disappointing result is a solver problem.
1. Freeze the task and evaluation evidence
Write the prediction unit, time, target horizon, feature-availability rule, positive class if relevant, and primary metric. Preserve row/entity ids, dataset provenance, exclusions, preprocessing state, and selection history. Do not improve the score by deleting difficult test cases while investigating. Check whether the reported value is training, inner validation, outer assessment, or a final untouched test. A column named test is not independent if engineers repeatedly used its errors to revise the system. Keep sensitive records private and build a small synthetic reproduction of the suspected boundary violation.
2. Audit entity overlap and temporal availability
Compare the identifiers actually used in each split, including household, device, site, or other dependency units that matter to deployment. The local example reveals one entity present on both sides; it does not assume every shared entity is always invalid, because a known-entity task can have another contract. Check feature and label availability timestamps separately. A row dated before a cutoff can still have a label finalized afterward. Correct group or time boundaries according to the intended use, preserving an honest final assessment instead of simply reshuffling until the score looks plausible.
train_entities = {"event-1", "event-2", "event-3"}
validation_entities = {"event-3", "event-4"}
print(sorted(train_entities & validation_entities))
['event-3']This complete local check reports overlap for the declared entity ids. Interpreting that overlap requires the task’s independence contract.
3. Trace every learned transform and selection step
Identify where means, imputation values, category vocabularies, feature screens, PCA bases, calibration mappings, and thresholds were fitted. Under ordinary held-out evaluation, each must obey its permitted training or calibration boundary. An unsupervised transform fitted on all features can still expose the validation distribution. Move those steps inside the appropriate fold or pipeline and inspect any earlier preprocessing outside it. Do not merely wrap a leaked array in a pipeline and call the procedure repaired. Verify target alignment by stable observation identity, because shuffled labels or duplicate joins can create another kind of invalid evidence.
4. Rebuild the faithful comparison and verify the boundary
Rerun a simple baseline and the full candidate under the repaired split, using the same metric and approved data. Expect the new number to differ; lower but valid evidence is not a regression in scientific honesty. Compare slices and sample counts, and document which prior result is invalidated. Add tests that prove train-only statistics, disjoint groups where required, and availability rules. If final-test outcomes were consumed during repair, establish an honest new assessment plan. The course’s local programs verify these checks only; they do not fit or evaluate any external dataset.
Verify the fix
The repaired procedure documents what each step may see, preserves required entity/time separation, and evaluates the whole pipeline rather than only its final estimator. A baseline and negative leakage checks pass under the same protocol. Report the new result with scope and uncertainty, without claiming that passing these checks proves every possible leakage source absent.
Keep it from recurring
- Version split manifests and feature-availability contracts, keep learned preprocessing inside selection boundaries, and record which data informed every revision. Protect an untouched final assessment and review joins, duplicates, and hidden entity relationships before fitting.