The Limits of AI Verification: Intent vs. Specification in Developer Workflows

Description and use

A developer explores the limits of AI in coding. AI excels at verifying code against specifications but fails to understand developer intent and design decisions. The primary risk is not replacement, but human complacency and the abandonment of verification due to perceived AI reliability. Humans must retain control and define what 'correct' means.

Problem

Developers face the risk of losing professional relevance as AI automates coding and verification tasks, potentially leading to a state where human oversight is abandoned due to convenience rather than necessity.

Outcome

The experiment revealed that AI excels at verifying code against specifications but fails to understand intent, design decisions, or the practical utility of a feature. The human's unique value lies in defining intent and maintaining rigorous oversight to prevent complacency.

Workflow

• Deploy AI agents to perform initial coding, diagnosis, and verification passes.
• Allow agents to refute each other's diagnoses and stop execution on false premises.
• Review AI-generated defect lists to distinguish between actual bugs and intended design decisions (intent verification).
• Verify that code matches the specification, but critically assess if the specification matches the user's actual intent.
• In domains outside core expertise, refuse to sign off on AI conclusions without rigorous pressure-testing and exact alignment of dependencies.
• Maintain active control and checking habits to prevent 'human degradation' and complacency.

Evidence

A verifier can prove a system does what it says. It cannot tell you whether what it says is what you meant.

Source

The seat doesn't disappear because something takes it. It disappears because sitting in it stops feeling necessary.

Source

I removed three of them on sight — not because the analysis was wrong, but because the behaviour was intended.

Source