Our Recall Was 0.087 and the Model Was Innocent: How Domain-Scoped Replay Doubled It

说明与用途

CauterRule通过按域限定参考轨迹，提升召回率，效果翻倍，未改变模型。

问题

CauterRule’s recall metric was inflated by including irrelevant failures in the denominator, causing good rules to appear worthless.

结果

Domain‑scoped replay increased recall by 2–3× across two cloud models, improved golden pass rates to 40–50%, and made recall diagnosable without changing the matcher or model.

工作流

• 1. Identify candidate rule’s source domain from its trajectory metadata.
• 2. Filter reference_pool to only trajectories with matching domain.
• 3. Count prevented failures where rule matches and trajectory failed.
• 4. Compute recall as prevented / (total failures in scoped pool).
• 5. Apply new pass threshold (0.5) to determine rule acceptance.

证据

The pool did not shrink — it grew from 230 to 444 references. What changed is the slice each candidate is judged against:

来源

Recall pre 0.068, recall post 0.170 (gpt‑4o‑mini) – 2.5× increase

来源

We made one more change alongside it, and it deserves to be stated plainly: the pass threshold dropped from 0.8 to 0.5.

来源