AI Coding Tools Are Great Until the Only Test Is a Device in Someone's Hand

Description and use

AI tools speed up boilerplate, config, and test scaffolding, yet need strict governance for niche APIs, editor-owned files, and device-dependent features.

Problem

AI coding tools perform well on projects with abundant public code and fast feedback loops, but their effectiveness drops sharply when applied to niche platform APIs, editor-owned file formats, or systems where the only real test is a physical device.

Outcome

AI tools accelerate boilerplate, config, test scaffolding, and codebase orientation, but require strict governance to avoid errors in platform-specific logic, editor-managed assets, and device-dependent behavior.

Workflow

• 1. Write a detailed spec outlining fields, tools, and rules.
• 2. Prompt Claude Code with the spec to generate code, configs, or tests.
• 3. Review the diff as if a junior engineer wrote it, checking logic and edge cases.
• 4. Verify changes on the real target (device, staging deploy, or real test call).
• 5. Iterate with small, focused changes, keeping the loop tight.

Evidence

AI coding tools are excellent where there is a lot of consistent public code and a fast way to check the result. They get worse as you move toward niche platform APIs, editor-owned file formats, and systems where the only real test is a device in someone's hand.

Source

Spec first, prompt second. If I cannot write down what done looks like, the tool cannot either.

Source

Review every generated change as if a new junior engineer wrote it.

Source

Verify on the real target. A device, a real test call, a staging deploy.

Source