I Shipped a Fix That Fixed Nothing. Here’s Why I Kept It

Description and use

The article explains how the CauterRule team built a gate to drop near‑miss trajectories that would otherwise generate false rules. The gate (Fix 6) had no impact on current data but serves as a future defense. The real improvement came from Fix 8 in the simulator, raising the golden pass rate from 20% to 50%.

Problem

A hypothesis that near‑miss false positives were caused by recovery patterns led to a gate that never fired on the current corpus.

Outcome

Fix 6 did not affect current metrics but provides a cheap, LLM‑free defense against future recovery trajectories. Fix 8 delivered measurable improvement by correcting a simulator classification bug.

Workflow

• 1. Identify near‑miss trajectories with success=True, early error, and later clean output.
• 2. Implement gate logic in gate.py to drop such trajectories before LLM call.
• 3. Run field test sweep with Fix 6 applied and observe zero drops.
• 4. Document that the hypothesis was wrong and the gate is a future‑proof defense.
• 5. Apply Fix 8 in the simulator to reclassify recovery trajectories as near_miss during replay.
• 6. Measure improvement in golden pass rate (20%→50%).

Evidence

Fix 6 dropped 0 trajectories in practice — the nearmiss corpus trajectories have success=False, not recovery patterns.

Source

Fix 8 moved golden 20% → 50% on both cloud models.

Source