How do you know if your coding agent's instructions are getting better?

Description and use

<p>A common way to improve a coding agent is to change its instructions.</p> <p>You add a rule to AGENTS.md. You write a skill for a recurring task. You tell the agent to check something before editing code.</p> <p>Then the next run looks better.</p> <p>But was the instruction responsible?</p> <p>That's a harder question than it first appears.</p> <h2> The problem with judging one successful run </h2> <p>Suppose an agent repeatedly makes an error. You add a new instruction explaining how to avoid it.</p> <p>The agent succeeds on the next attempt.</p> <p>That result is encouraging, but it doesn

Problem

<p>A common way to improve a coding agent is to change its instructions.</p> <p>You add a rule to AGENTS.md. You write a skill for a recurring task. You tell the agent to check something before editing code.</p> <p>Then the next run looks better.</p> <p>But was the instruction responsible?</p> <p>That's a harder question than it first appears.</p> <h2> The problem with judging one successful run </h2> <p>Suppose an agent repeatedly makes an error. You add a new instruction explaining how to avoid it.</p> <p>The agent succeeds on the next attempt.</p> <p>That result is encouraging, but it doesn

Outcome

<p>A common way to improve a coding agent is to change its instructions.</p> <p>You add a rule to AGENTS.md. You write a skill for a recurring task. You tell the agent to check something before editing code.</p> <p>Then the next run looks better.</p> <p>But was the instruction responsible?</p> <p>That's a harder question than it first appears.</p> <h2> The problem with judging one successful run </h2> <p>Suppose an agent repeatedly makes an error. You add a new instruction explaining how to avoid it.</p> <p>The agent succeeds on the next attempt.</p> <p>That result is encouraging, but it doesn

Evidence

How do you know if your coding agent's instructions are getting better?

Source