I Shipped a Fix That Fixed Nothing. Here’s Why I Kept It

Описание и применение

В статье описывается, как команда CauterRule создала защиту от «приближённых» ошибок, которая не изменила текущие метрики, но обеспечивает защиту от будущих сценариев. Реальная выгода пришла от исправления в симуляторе, которое повысило процент успешных тестов с 20% до 50%.

Задача

A hypothesis that near‑miss false positives were caused by recovery patterns led to a gate that never fired on the current corpus.

Результат

Fix 6 did not affect current metrics but provides a cheap, LLM‑free defense against future recovery trajectories. Fix 8 delivered measurable improvement by correcting a simulator classification bug.

Рабочий процесс

• 1. Identify near‑miss trajectories with success=True, early error, and later clean output.
• 2. Implement gate logic in gate.py to drop such trajectories before LLM call.
• 3. Run field test sweep with Fix 6 applied and observe zero drops.
• 4. Document that the hypothesis was wrong and the gate is a future‑proof defense.
• 5. Apply Fix 8 in the simulator to reclassify recovery trajectories as near_miss during replay.
• 6. Measure improvement in golden pass rate (20%→50%).

Доказательства

Fix 6 dropped 0 trajectories in practice — the nearmiss corpus trajectories have success=False, not recovery patterns.

Источник

Fix 8 moved golden 20% → 50% on both cloud models.

Источник