A Drift Detector That Cries Wolf Is Worse Than No Detector

说明与用途

HivePlane v0.2.0 的漂移检测器因基准增长而错误隔离健康代理，错误地比较总失败计数。修复包括基准感知比较、基于隔离记录的恢复以及探针输出验证。

问题

A drift detector quarantined a healthy agent because it compared failure counts across growing benchmark sizes, treating benchmark growth as decay.

结果

Implemented baseline‑aware drift detection, corrected reinstatement logic, and enhanced probe to detect output decay, eliminating false positives and ensuring reliable quarantine control.

工作流

• 1. Run agent on current corpus. 2. Compare failure counts to previous baseline on same tasks. 3. If failures exceed threshold, auto‑quarantine. 4. Reinstatement checks quarantine record, not status enum. 5. Probe runner executes run, waits for terminal, compares output to expected; on mismatch, emits probe.decay warning.
• 6. Field tests (S4, S5, S8, S24) validate drift detection, quarantine, reinstatement, and probe decay detection.

证据

My drift detector quarantined a healthy agent. The agent hadn't gotten worse — the benchmark had gotten bigger.

来源

The detector compared failure counts across corpus versions, so growing the benchmark auto‑quarantined a healthy agent.

来源

The fix: drift is measured against the agent's own baseline on comparable corpora. You compare the same tasks across re‑evaluations, not different counts.

来源

The probe runner only checked admission. So a probe could only fail if the run was refused — it could never flag that the agent ran and produced the wrong answer.

来源

The runner starts the run, waits for terminal, and compares the output to expected. A mismatch fails the probe with a probe.decay warning.

来源