An AI's “Done” Can't Be Trusted — Quality Gates That Stop False Completion With Evidence

Описание и применение

Автоматизированные проверки качества, основанные на реальных доказательствах, устраняют ложные сообщения об завершении кода, генерируемого ИИ, и предотвращают ошибки в продакшене.

Задача

AI-generated code often reports completion based on internal signals (e.g., green unit tests) while real-world functionality remains broken, leading to false confidence and production failures.

Результат

By shifting from AI self‑reporting to machine‑verifiable evidence, the team eliminated false completion reports, caught packaging and deployment defects early, and prevented 12 days of distribution drift.

Рабочий процесс

• 1. Add a gate that blocks completion claims lacking evidence (e.g., only unit test success).
• 2. Require real‑device E2E evidence: API baseURL wired, auth flow tested, major flows manually run, SafeArea verified via screenshot.
• 3. Create a dedicated completion checklist for boundary‑heavy areas (mobile, payments, DB writes).
• 4. Run tests in a clean worktree to catch non‑hermetic tests.
• 5. Add a packaging contract test to ensure runtime dependencies (e.g., json‑repair) are present in the production image.
• 6. Implement a pull‑based parity check: a cron job on the distribution target reconciles the installed version against the expected version to detect deployment drift.

Доказательства

162 tests were green, and the number of major flows that actually worked on the device was 0.

Источник

json‑repair must be declared in the worker's requirements

Источник

A pull‑based parity check run at the distribution target detected 12 days of drift

Источник