Our Recall Was 0.087 and the Model Was Innocent: How Domain-Scoped Replay Doubled It

Описание и применение

CauterRule улучшил показатель recall, ограничив набор ссылок по домену, что удвоило эффективность правил без изменения модели.

Задача

CauterRule’s recall metric was inflated by including irrelevant failures in the denominator, causing good rules to appear worthless.

Результат

Domain‑scoped replay increased recall by 2–3× across two cloud models, improved golden pass rates to 40–50%, and made recall diagnosable without changing the matcher or model.

Рабочий процесс

• 1. Identify candidate rule’s source domain from its trajectory metadata.
• 2. Filter reference_pool to only trajectories with matching domain.
• 3. Count prevented failures where rule matches and trajectory failed.
• 4. Compute recall as prevented / (total failures in scoped pool).
• 5. Apply new pass threshold (0.5) to determine rule acceptance.

Доказательства

The pool did not shrink — it grew from 230 to 444 references. What changed is the slice each candidate is judged against:

Источник

Recall pre 0.068, recall post 0.170 (gpt‑4o‑mini) – 2.5× increase

Источник

We made one more change alongside it, and it deserves to be stated plainly: the pass threshold dropped from 0.8 to 0.5.

Источник