A Drift Detector That Cries Wolf Is Worse Than No Detector

Description and use

The HivePlane v0.2.0 drift detector mistakenly quarantined a healthy agent due to benchmark growth, comparing total failure counts. Fixes include baseline‑aware comparison, quarantine‑record‑based reinstatement, and probe output validation.

Problem

A drift detector quarantined a healthy agent because it compared failure counts across growing benchmark sizes, treating benchmark growth as decay.

Outcome

Implemented baseline‑aware drift detection, corrected reinstatement logic, and enhanced probe to detect output decay, eliminating false positives and ensuring reliable quarantine control.

Workflow

• 1. Run agent on current corpus. 2. Compare failure counts to previous baseline on same tasks. 3. If failures exceed threshold, auto‑quarantine. 4. Reinstatement checks quarantine record, not status enum. 5. Probe runner executes run, waits for terminal, compares output to expected; on mismatch, emits probe.decay warning.
• 6. Field tests (S4, S5, S8, S24) validate drift detection, quarantine, reinstatement, and probe decay detection.

Evidence

My drift detector quarantined a healthy agent. The agent hadn't gotten worse — the benchmark had gotten bigger.

Source

The detector compared failure counts across corpus versions, so growing the benchmark auto‑quarantined a healthy agent.

Source

The fix: drift is measured against the agent's own baseline on comparable corpora. You compare the same tasks across re‑evaluations, not different counts.

Source

The probe runner only checked admission. So a probe could only fail if the run was refused — it could never flag that the agent ran and produced the wrong answer.

Source

The runner starts the run, waits for terminal, and compares the output to expected. A mismatch fails the probe with a probe.decay warning.

Source