A Drift Detector That Cries Wolf Is Worse Than No Detector

Описание и применение

Детектор дрейфа в HivePlane v0.2.0 ошибочно изолировал здорового агента из‑за роста набора задач, сравнивая общее число неудач. Исправления включают сравнение на одинаковых задачах, использование записи о карантине для восстановления и проверку ожидаемого вывода в пробе.

Задача

A drift detector quarantined a healthy agent because it compared failure counts across growing benchmark sizes, treating benchmark growth as decay.

Результат

Implemented baseline‑aware drift detection, corrected reinstatement logic, and enhanced probe to detect output decay, eliminating false positives and ensuring reliable quarantine control.

Рабочий процесс

• 1. Run agent on current corpus. 2. Compare failure counts to previous baseline on same tasks. 3. If failures exceed threshold, auto‑quarantine. 4. Reinstatement checks quarantine record, not status enum. 5. Probe runner executes run, waits for terminal, compares output to expected; on mismatch, emits probe.decay warning.
• 6. Field tests (S4, S5, S8, S24) validate drift detection, quarantine, reinstatement, and probe decay detection.

Доказательства

My drift detector quarantined a healthy agent. The agent hadn't gotten worse — the benchmark had gotten bigger.

Источник

The detector compared failure counts across corpus versions, so growing the benchmark auto‑quarantined a healthy agent.

Источник

The fix: drift is measured against the agent's own baseline on comparable corpora. You compare the same tasks across re‑evaluations, not different counts.

Источник

The probe runner only checked admission. So a probe could only fail if the run was refused — it could never flag that the agent ran and produced the wrong answer.

Источник

The runner starts the run, waits for terminal, and compares the output to expected. A mismatch fails the probe with a probe.decay warning.

Источник