Testing AI workflow automation on a tender: a cite check with a planted answer key

说明与用途

使用植入答案键测试招标文件的 AI 工作流自动化，以衡量引用检查的准确性。

问题

How to validate an AI-powered document check before trusting it, ensuring accurate flagging of citation errors.

结果

With annexes, both models caught at least nine of ten errors; Sonnet achieved 58/60 catches, Haiku 54/60. The skill added little benefit. Tender‑only runs showed Sonnet could not check without sources, while Haiku flagged 13 errors.

工作流

• Create a synthetic tender with 26 claims and 10 planted citation errors.
• Write pack.py to validate claim references and counts.
• Define a single tool schema (submit_review) in api_job.py.
• Run six test conditions per model (tender only, tender+annexes, annexes+skill).
• Score results with score.py, matching caught claims to planted errors.
• Analyze false alarms and missed errors.
• Iterate plumbing fixes to enable proper tool usage.
• Document performance metrics for Sonnet 5.5 and Haiku 5.5.

证据

The request is identical in every run, in Danish, from an invented sales lead: here is section 4 and the annexes it cites, it goes out Friday at 12, can you read it through and say what needs fixing before we send?

来源

Two faults, both the builder's: - It forced the tool call with tool_choice. Claude Sonnet 5.5 does not take a forced tool call, so all 12 Sonnet runs came back as a 400 error and no Sonnet review exists from that run.

来源

With the annexes, every run caught at least nine of ten, and there was one false alarm in 24 runs: Sonnet asked for a source for the figure of 60 technicians in claim 4.1.

来源