LLM Order-Reading: From Exam Passing to Production Safety

Description and use

A developer describes deploying an LLM for order reading. Despite passing an internal exam, the system isn't ready for release due to lack of real-world data. The proposed workflow involves initial human review, converting errors into new tests, and safely deferring ambiguous cases to humans.

Problem

An LLM passed a 29-question internal exam for order reading with zero fatal errors, but shipping it immediately is risky because the exam only covers scenarios the developer could imagine, leaving gaps for real-world edge cases like ambiguous abbreviations or dirty product catalogs.

Outcome

The workflow shifted from trusting static exam scores to a dynamic, production-driven testing loop. The exam caught more bugs in the grading logic and test data than in the model itself. The key outcome is a safe deployment strategy that prioritizes human review for unknown cases, preventing wrong goods from being shipped.

Workflow

• Create a synthetic exam covering known worst-case scenarios.
• Verify the model against the exam, ensuring zero fatal errors.
• Implement a safety guard: if the model is unsure, it must hand off to a human instead of guessing.
• Launch with a 'watch period' where humans review every result.
• Convert every human-corrected miss into a new exam question.
• After 1-4 weeks, if fatal errors remain at zero, enable auto-pass for confirmed cases.
• Transition to a mode where humans only review the 'needs confirmation' queue.
• Identify impossible questions (e.g., interpreting 'the usual') and shrink scope to exclude them from automation.

Evidence

The model's real mistakes — 1. And its only crime was asking. The exam author's mistakes — 5. Three in the answer key, two in the grader.

Source

If the unknown case falls somewhere safe by construction, an incomplete exam is still shippable.

Source