Human-in-the-loop: designing the review queue

Описание и применение

Дизайн очереди проверки с участием человека, где учитываются низкая уверенность и жесткие правила, рассчитывается потребность в рецензентах, предоставляется все доказательство на одном экране, отслеживается коэффициент переопределения и проверяется качество рецензентов с помощью семпл‑проверок.

Задача

Review queues that are slow, confusing, or endless erode trust, leading to rubber‑stamping and loss of accuracy.

Результат

The queue design improves trust, reduces review time, provides measurable metrics (override rate, reviewer hours), and turns corrections into training data for continuous model improvement.

Рабочий процесс

• Identify items below confidence threshold or that hit hard rules
• Record reason for queue entry
• Calculate reviewer hours using items_per_day × share_to_review × minutes_per_review / 60
• Present item, proposed answer, reason, evidence, and one‑click actions on a single screen
• Reviewers accept, correct, or reject; corrections are stored as labeled examples
• Monthly re‑tune threshold and address top confusions
• Track override rate and seed known‑answer items to monitor reviewer diligence

Доказательства

If the review queue is slow, confusing or endless, people stop trusting it, start rubber‑stamping, and the accuracy you measured in the pilot quietly disappears.

Источник

Two kinds of items belong there: Low confidence. Anything below the threshold you agreed on. Hard rules. Items that must always be checked regardless of confidence.

Источник

items_per_day × share_to_review × minutes_per_review / 60 = reviewer_hours_per_day

Источник

Show the reviewer what they need in one screen: The item itself, and the system's proposed answer. Why it is in the queue: low confidence, which rule fired. The evidence: the cited passage, the extracted field highlighted in the source, the runner‑up answer. One‑click actions: accept, correct, reject.

Источник

Seed a small share of known‑answer items into the queue and measure whether they are caught.

Источник