No agent grades its own homework

Описание и применение

Когда LLM‑агент проверяет собственный код, он повторяет свои ошибки. В Agateon решение – разделить роли: оркестратор назначает работу, новый подагент реализует её, а независимый судья проверяет результат в чистом контексте, используя машинный exit‑код. Это устраняет «эхо» и повышает надёжность.

Задача

When an LLM agent reviews its own output it echoes its own misunderstandings, missing bugs and misinterpretations.

Результат

The new structure prevented two bad commits that existed only in logs, caught a writer agent’s fabricated anecdote, and ensured that verification was independent of author context, improving reliability.

Рабочий процесс

• 1. Orchestrator reads state and dispatches a phase to a fresh subagent with only the brief and protocol files.
• 2. Subagent implements the task and writes output to a read‑only artifact.
• 3. Judge agent is spawned in a brand‑new session, receives only the artifact and spec, and runs automated tests, producing an exit code.
• 4. Acceptance gate checks the exit code; if it passes, the artifact is committed; if not, the orchestrator retries or aborts.
• 5. All retries and outcomes are logged with round number, failure mode, and prompt changes for auditability.

Доказательства

We ran our whole engineering workflow this way, and it has caught things the author‑agent could not have caught: two bad commits that exist only in a log and never in git, an implementation sent back twice by a judge that had never seen our reasoning, and — a few days ago — a blog draft where a writer agent invented a personal anecdote to sound more human, which a fresh‑context reviewer flagged while the writer never noticed.

Источник

The orchestrator never touches the artifact. It reads state, dispatches phases, and runs gate commands — read‑only verification, never file edits.

Источник