Testing AI workflow automation on a tender: a cite check with a planted answer key

Description and use

Testing AI workflow automation on a tender with a planted answer key to measure citation‑check accuracy.

Problem

How to validate an AI-powered document check before trusting it, ensuring accurate flagging of citation errors.

Outcome

With annexes, both models caught at least nine of ten errors; Sonnet achieved 58/60 catches, Haiku 54/60. The skill added little benefit. Tender‑only runs showed Sonnet could not check without sources, while Haiku flagged 13 errors.

Workflow

• Create a synthetic tender with 26 claims and 10 planted citation errors.
• Write pack.py to validate claim references and counts.
• Define a single tool schema (submit_review) in api_job.py.
• Run six test conditions per model (tender only, tender+annexes, annexes+skill).
• Score results with score.py, matching caught claims to planted errors.
• Analyze false alarms and missed errors.
• Iterate plumbing fixes to enable proper tool usage.
• Document performance metrics for Sonnet 5.5 and Haiku 5.5.

Evidence

The request is identical in every run, in Danish, from an invented sales lead: here is section 4 and the annexes it cites, it goes out Friday at 12, can you read it through and say what needs fixing before we send?

Source

Two faults, both the builder's: - It forced the tool call with tool_choice. Claude Sonnet 5.5 does not take a forced tool call, so all 12 Sonnet runs came back as a 400 error and no Sonnet review exists from that run.

Source

With the annexes, every run caught at least nine of ten, and there was one false alarm in 24 runs: Sonnet asked for a source for the figure of 60 technicians in claim 4.1.

Source