Trust, but Verify: How We Audited OpenAI's Lean 4 Math Proofs with a Zero-Dependency Sidecar

Описание и применение

Автоматизированный аудит Lean 4 доказательств OpenAI выявил 11 корректных доказательств и 12 шаблонов с незавершёнными «sorry»‑заполнителями, демонстрируя эффективность физического‑доказательского подхода.

Задача

Verifying thousands of lines of formal Lean 4 proofs claimed by an AI lab without expert human review or hallucinating LLM judges.

Результат

The audit confirmed 11 main proofs were valid and 12 template files contained unproved placeholders, demonstrating the sidecar’s ability to detect incomplete proofs that an LLM judge would miss.

Рабочий процесс

• Clone OpenAI ten-proofs repo
• Pull 7.2 GB Mathlib cache
• Compile 8,687 Lean jobs locally
• Run LeanProofAuditor from harness-audit-sidecar
• Collect four-valued verdicts (PASS, FAIL, INVALID, NOT RUN)
• Report 11 PASS files and 12 FAIL files with 42 sorry placeholders

Доказательства

The audit took about 40 minutes. Here are the hard facts:

Источник

The 11 Main Proof Files achieved a genuine PASS. All theorems elaborated successfully. Zero smuggled custom axioms.

Источник

12 Files returned FAIL (42 placeholders found). These 12 files are located in ComparatorChallenges/. They are templates intentionally left with sorry for human contestants to fill out.

Источник

Our sidecar enforces a four-valued verdict: PASS, FAIL, INVALID, NOT RUN.

Источник