A Terminal-Bench task frontier agents could not pass

Description and use

Agents failed to correctly redact names and IDs in PDFs, leaving hidden data and over‑redacting. The task split rules and hidden cases, yet all attempts failed.

Problem

Agents failed to correctly redact court‑filing PDFs, leaving hidden names and IDs while over‑redacting other content.

Outcome

All agent attempts failed; the task remains a hard benchmark for PDF redaction.

Workflow

• Define policy with all redaction rules.
• Provide three sample PDF files to agents.
• Run agents with highest reasoning settings, allowing them to build tools and generate tests.
• Freeze and checksum the task before trials.
• Run one normal trial and one cheat trial first, then the rest.
• Reject any run that hits harness errors, rate limits, or probes the grader.
• Audit agent transcripts to ensure sandbox isolation.
• Iterate on task design until reference solution passes and do‑nothing fails.
• Publish final task and evidence on GitHub.

Evidence

All six runs failed, and so did two runs where the agent was told to cheat.

Source

The policy states every rule in full. The hidden test set includes cases no sample shows, and each one is decided by a rule the policy already states.

Source

The reference solution scores 100% and a do‑nothing submission scores 0% before any agent runs.

Source

I also audited the transcripts of the agents I used while building the task, to make sure none of them reached outside their sandbox.

Source