Human-in-the-loop: designing the review queue

说明与用途

设计人机协作的审核队列，区分低置信度和硬规则项目，计算审核员容量，单屏展示所有证据，跟踪覆盖率，并通过种子已知答案项目防止人机刷单。

问题

Review queues that are slow, confusing, or endless erode trust, leading to rubber‑stamping and loss of accuracy.

结果

The queue design improves trust, reduces review time, provides measurable metrics (override rate, reviewer hours), and turns corrections into training data for continuous model improvement.

工作流

• Identify items below confidence threshold or that hit hard rules
• Record reason for queue entry
• Calculate reviewer hours using items_per_day × share_to_review × minutes_per_review / 60
• Present item, proposed answer, reason, evidence, and one‑click actions on a single screen
• Reviewers accept, correct, or reject; corrections are stored as labeled examples
• Monthly re‑tune threshold and address top confusions
• Track override rate and seed known‑answer items to monitor reviewer diligence

证据

If the review queue is slow, confusing or endless, people stop trusting it, start rubber‑stamping, and the accuracy you measured in the pilot quietly disappears.

来源

Two kinds of items belong there: Low confidence. Anything below the threshold you agreed on. Hard rules. Items that must always be checked regardless of confidence.

来源

items_per_day × share_to_review × minutes_per_review / 60 = reviewer_hours_per_day

来源

Show the reviewer what they need in one screen: The item itself, and the system's proposed answer. Why it is in the queue: low confidence, which rule fired. The evidence: the cited passage, the extracted field highlighted in the source, the runner‑up answer. One‑click actions: accept, correct, reject.

来源

Seed a small share of known‑answer items into the queue and measure whether they are caught.

来源