An AI's “Done” Can't Be Trusted — Quality Gates That Stop False Completion With Evidence

说明与用途

基于真实证据的机器强制质量门控消除了 AI 生成代码的虚假完成报告，防止生产缺陷。

问题

AI-generated code often reports completion based on internal signals (e.g., green unit tests) while real-world functionality remains broken, leading to false confidence and production failures.

结果

By shifting from AI self‑reporting to machine‑verifiable evidence, the team eliminated false completion reports, caught packaging and deployment defects early, and prevented 12 days of distribution drift.

工作流

• 1. Add a gate that blocks completion claims lacking evidence (e.g., only unit test success).
• 2. Require real‑device E2E evidence: API baseURL wired, auth flow tested, major flows manually run, SafeArea verified via screenshot.
• 3. Create a dedicated completion checklist for boundary‑heavy areas (mobile, payments, DB writes).
• 4. Run tests in a clean worktree to catch non‑hermetic tests.
• 5. Add a packaging contract test to ensure runtime dependencies (e.g., json‑repair) are present in the production image.
• 6. Implement a pull‑based parity check: a cron job on the distribution target reconciles the installed version against the expected version to detect deployment drift.

证据

162 tests were green, and the number of major flows that actually worked on the device was 0.

来源

json‑repair must be declared in the worker's requirements

来源

A pull‑based parity check run at the distribution target detected 12 days of drift

来源