Your AI Agent Says “Done.” What Did It Actually Verify?

Description and use

Verifying AI agent completion: ensuring a bug fix truly works and preserves other behavior.

Problem

When an AI agent reports completion, it is unclear whether the intended behavior was truly achieved and verified.

Outcome

The workflow ensures that an AI agent’s claim of completion is backed by concrete, verifiable evidence, reducing the risk of merging incomplete or incorrect changes.

Workflow

• Define observable "done" criteria before the agent starts.
• Ask the agent to include evidence in its completion report.
• Ensure the test catches the original bug by verifying expected outcomes.
• Inspect any changes made to the tests themselves.
• Match verification methods to the type of change (calculation, API, UI, etc.).
• Check for unintended side effects or scope creep in the diff.
• Treat blocked verification as unfinished and document missing checks.
• Use a reusable prompt to standardize reporting and verification.
• Ask for evidence that the change meets the requirement before merging.

Evidence

"Fixed the bug. All tests pass."

Source

"Added a regression test for excessive discounts. The checkout test suite passed. Browser checkout was not tested because payment credentials were unavailable."

Source

When a discount exceeds the order subtotal, the total should be zero. Add a regression test. Preserve valid discount behavior.

Source