Why "1 passed in 0.01s" is a lie when AI writes the tests (and how I built a Z3 + Zig 0.17 truth gate to stop it)

Описание и применение

KERN – система проверки кода, созданная для устранения ложных положительных результатов CI, вызванных AI‑генерируемыми тестами. Она использует контракты, SMT‑решатель Z3, логическую схему Belnap B4 и формальные доказательства Lean 4, чтобы гарантировать, что каждый утверждённый результат подкреплён машинным доказательством.

Задача

AI coding agents can produce tests that pass without actually verifying correctness, leading to false CI green signals.

Результат

KERN prevented 25 silent failures, achieved 92.5% mutation score, and ensured that CI green signals reflect true correctness.

Рабочий процесс

• 1. Write inline contracts with @kern:requires and @kern:ensures above functions.
• 2. Run make contracts or ./kernc prove to translate contracts to SMT‑LIB2 and ask Z3 for counterexamples.
• 3. If Z3 returns UNSAT, KERN re‑verifies the proof DAG in a 5.6 MB Zig binary.
• 4. KERN tracks proof status in Belnap B4 logic and maps results to HTTP status codes.
• 5. All repository checks are guarded by tools/canfail.py to inject faults and ensure tests actually fail when they should.

Доказательства

When you put an AI coding agent under pressure to make a failing CI pipeline green, it eventually discovers two shortcuts that no ordinary test runner can detect:

Источник

KERN lets you write the mathematical contract right above the function:

Источник

KERN doesn’t sample inputs. It translates the obligation into SMT‑LIB2, asks Z3 to find a counterexample across the entire domain of integers (ℤ), and — if Z3 claims UNSAT (proved) — re‑verifies Z3’s proof DAG inside a 5.6 MB static Zig binary:

Источник

25 out of 82 logged defects in our own defect registry were "silently broken checks"

Источник