A Terminal-Bench task frontier agents could not pass

说明与用途

代理无法正确删除PDF中的姓名和ID，留下隐藏数据并过度编辑。任务将规则和隐藏案例分离，但所有尝试均失败。

问题

Agents failed to correctly redact court‑filing PDFs, leaving hidden names and IDs while over‑redacting other content.

结果

All agent attempts failed; the task remains a hard benchmark for PDF redaction.

工作流

• Define policy with all redaction rules.
• Provide three sample PDF files to agents.
• Run agents with highest reasoning settings, allowing them to build tools and generate tests.
• Freeze and checksum the task before trials.
• Run one normal trial and one cheat trial first, then the rest.
• Reject any run that hits harness errors, rate limits, or probes the grader.
• Audit agent transcripts to ensure sandbox isolation.
• Iterate on task design until reference solution passes and do‑nothing fails.
• Publish final task and evidence on GitHub.

证据

All six runs failed, and so did two runs where the agent was told to cheat.

来源

The policy states every rule in full. The hidden test set includes cases no sample shows, and each one is decided by a rule the policy already states.

来源

The reference solution scores 100% and a do‑nothing submission scores 0% before any agent runs.

来源

I also audited the transcripts of the agents I used while building the task, to make sure none of them reached outside their sandbox.

来源