Multirun: A Multi-Agent Run Framework

说明与用途

多次独立运行代理程序可揭示模型变异性，帮助评估并做出更可靠的决策，降低错误成本。

问题

Running a coding agent once gives only an anecdote; to understand model variability and avoid blind decisions, multiple isolated runs are needed.

结果

Ten parallel runs revealed that all produced the same schema but split 5/5 on two design decisions and 8/2 on a NULL constraint, demonstrating model variability. The framework enabled cheap evaluation and informed branching decisions, saving potential costly mistakes.

工作流

• 1. Define source repository at a specific SHA and optional database seed.
• 2. Create a task as a markdown prompt with checks, versioned as files.
• 3. Configure a fan‑out run: set global defaults and per‑run overrides (here all runs identical).
• 4. For each run, create a git worktree and a Postgres branch off the same parent.
• 5. Execute the agent (Claude Code via Anthropic Agent SDK) on the task in the isolated environment.
• 6. Capture artifacts (code, transcripts, diffs).
• 7. Diff each run’s branch back to the parent to obtain schema and data deltas.
• 8. Teardown the branch after diffing to avoid orphaned resources.
• 9. Store run state and artifacts in a Postgres results database for queryability.
• 10. Review the spread of outputs to inform evaluation or branching decisions.

证据

Ten runs in this fan‑out took fifty minutes of wall time and about thirty‑two dollars at API rates

来源

All ten reached the same schema shape. Then they split 5/5 on whether deleting a team deletes its data, 5/5 on the join table's primary key, and 8/2 on whether the new column is allowed to be NULL.

来源

Ten runs need ten databases, and they all need to begin life identical.

来源