I Shipped a Fix That Fixed Nothing. Here’s Why I Kept It

说明与用途

文章说明了 CauterRule 团队如何构建一个门控机制来丢弃近似错误轨迹，从而防止生成错误规则。该门控（Fix 6）对当前数据没有影响，但为未来数据提供防御。真正的改进来自模拟器中的 Fix 8，将黄金通过率从 20% 提升到 50%。

问题

A hypothesis that near‑miss false positives were caused by recovery patterns led to a gate that never fired on the current corpus.

结果

Fix 6 did not affect current metrics but provides a cheap, LLM‑free defense against future recovery trajectories. Fix 8 delivered measurable improvement by correcting a simulator classification bug.

工作流

• 1. Identify near‑miss trajectories with success=True, early error, and later clean output.
• 2. Implement gate logic in gate.py to drop such trajectories before LLM call.
• 3. Run field test sweep with Fix 6 applied and observe zero drops.
• 4. Document that the hypothesis was wrong and the gate is a future‑proof defense.
• 5. Apply Fix 8 in the simulator to reclassify recovery trajectories as near_miss during replay.
• 6. Measure improvement in golden pass rate (20%→50%).

证据

Fix 6 dropped 0 trajectories in practice — the nearmiss corpus trajectories have success=False, not recovery patterns.

来源

Fix 8 moved golden 20% → 50% on both cloud models.

来源