Trust, but Verify: How We Audited OpenAI's Lean 4 Math Proofs with a Zero-Dependency Sidecar

说明与用途

对 OpenAI Lean 4 证明的自动化审计发现 11 个有效证明和 12 个包含 42 个 "sorry" 占位符的不完整模板，展示了物理证人验证方法的有效性。

问题

Verifying thousands of lines of formal Lean 4 proofs claimed by an AI lab without expert human review or hallucinating LLM judges.

结果

The audit confirmed 11 main proofs were valid and 12 template files contained unproved placeholders, demonstrating the sidecar’s ability to detect incomplete proofs that an LLM judge would miss.

工作流

• Clone OpenAI ten-proofs repo
• Pull 7.2 GB Mathlib cache
• Compile 8,687 Lean jobs locally
• Run LeanProofAuditor from harness-audit-sidecar
• Collect four-valued verdicts (PASS, FAIL, INVALID, NOT RUN)
• Report 11 PASS files and 12 FAIL files with 42 sorry placeholders

证据

The audit took about 40 minutes. Here are the hard facts:

来源

The 11 Main Proof Files achieved a genuine PASS. All theorems elaborated successfully. Zero smuggled custom axioms.

来源

12 Files returned FAIL (42 placeholders found). These 12 files are located in ComparatorChallenges/. They are templates intentionally left with sorry for human contestants to fill out.

来源

Our sidecar enforces a four-valued verdict: PASS, FAIL, INVALID, NOT RUN.

来源