Why "1 passed in 0.01s" is a lie when AI writes the tests (and how I built a Z3 + Zig 0.17 truth gate to stop it)

Description and use

KERN is a verification system that eliminates false CI positives caused by AI‑generated tests. It uses inline contracts, the Z3 SMT solver, Belnap B4 logic, and Lean 4 proofs to ensure every claim is backed by machine evidence.

Problem

AI coding agents can produce tests that pass without actually verifying correctness, leading to false CI green signals.

Outcome

KERN prevented 25 silent failures, achieved 92.5% mutation score, and ensured that CI green signals reflect true correctness.

Workflow

• 1. Write inline contracts with @kern:requires and @kern:ensures above functions.
• 2. Run make contracts or ./kernc prove to translate contracts to SMT‑LIB2 and ask Z3 for counterexamples.
• 3. If Z3 returns UNSAT, KERN re‑verifies the proof DAG in a 5.6 MB Zig binary.
• 4. KERN tracks proof status in Belnap B4 logic and maps results to HTTP status codes.
• 5. All repository checks are guarded by tools/canfail.py to inject faults and ensure tests actually fail when they should.

Evidence

When you put an AI coding agent under pressure to make a failing CI pipeline green, it eventually discovers two shortcuts that no ordinary test runner can detect:

Source

KERN lets you write the mathematical contract right above the function:

Source

KERN doesn’t sample inputs. It translates the obligation into SMT‑LIB2, asks Z3 to find a counterexample across the entire domain of integers (ℤ), and — if Z3 claims UNSAT (proved) — re‑verifies Z3’s proof DAG inside a 5.6 MB static Zig binary:

Source

25 out of 82 logged defects in our own defect registry were "silently broken checks"

Source