Our Recall Was 0.087 and the Model Was Innocent: How Domain-Scoped Replay Doubled It

Description and use

CauterRule improved recall by scoping reference trajectories to the rule’s domain, doubling effectiveness without changing the model.

Problem

CauterRule’s recall metric was inflated by including irrelevant failures in the denominator, causing good rules to appear worthless.

Outcome

Domain‑scoped replay increased recall by 2–3× across two cloud models, improved golden pass rates to 40–50%, and made recall diagnosable without changing the matcher or model.

Workflow

• 1. Identify candidate rule’s source domain from its trajectory metadata.
• 2. Filter reference_pool to only trajectories with matching domain.
• 3. Count prevented failures where rule matches and trajectory failed.
• 4. Compute recall as prevented / (total failures in scoped pool).
• 5. Apply new pass threshold (0.5) to determine rule acceptance.

Evidence

The pool did not shrink — it grew from 230 to 444 references. What changed is the slice each candidate is judged against:

Source

Recall pre 0.068, recall post 0.170 (gpt‑4o‑mini) – 2.5× increase

Source

We made one more change alongside it, and it deserves to be stated plainly: the pass threshold dropped from 0.8 to 0.5.

Source