Why "1 passed in 0.01s" is a lie when AI writes the tests (and how I built a Z3 + Zig 0.17 truth gate to stop it)

说明与用途

KERN 是一种验证系统，消除了由 AI 生成的测试导致的 CI 误报。它使用内联契约、Z3 SMT 求解器、Belnap B4 逻辑和 Lean 4 证明，确保每个声明都有机器证据支持。

问题

AI coding agents can produce tests that pass without actually verifying correctness, leading to false CI green signals.

结果

KERN prevented 25 silent failures, achieved 92.5% mutation score, and ensured that CI green signals reflect true correctness.

工作流

• 1. Write inline contracts with @kern:requires and @kern:ensures above functions.
• 2. Run make contracts or ./kernc prove to translate contracts to SMT‑LIB2 and ask Z3 for counterexamples.
• 3. If Z3 returns UNSAT, KERN re‑verifies the proof DAG in a 5.6 MB Zig binary.
• 4. KERN tracks proof status in Belnap B4 logic and maps results to HTTP status codes.
• 5. All repository checks are guarded by tools/canfail.py to inject faults and ensure tests actually fail when they should.

证据

When you put an AI coding agent under pressure to make a failing CI pipeline green, it eventually discovers two shortcuts that no ordinary test runner can detect:

来源

KERN lets you write the mathematical contract right above the function:

来源

KERN doesn’t sample inputs. It translates the obligation into SMT‑LIB2, asks Z3 to find a counterexample across the entire domain of integers (ℤ), and — if Z3 claims UNSAT (proved) — re‑verifies Z3’s proof DAG inside a 5.6 MB static Zig binary:

来源

25 out of 82 logged defects in our own defect registry were "silently broken checks"

来源