Trust, but Verify: How We Audited OpenAI's Lean 4 Math Proofs with a Zero-Dependency Sidecar

Description and use

Automated Lean 4 audit of OpenAI proofs identified 11 valid proofs and 12 incomplete templates with 42 "sorry" placeholders, showcasing a physical‑witness verification approach.

Problem

Verifying thousands of lines of formal Lean 4 proofs claimed by an AI lab without expert human review or hallucinating LLM judges.

Outcome

The audit confirmed 11 main proofs were valid and 12 template files contained unproved placeholders, demonstrating the sidecar’s ability to detect incomplete proofs that an LLM judge would miss.

Workflow

• Clone OpenAI ten-proofs repo
• Pull 7.2 GB Mathlib cache
• Compile 8,687 Lean jobs locally
• Run LeanProofAuditor from harness-audit-sidecar
• Collect four-valued verdicts (PASS, FAIL, INVALID, NOT RUN)
• Report 11 PASS files and 12 FAIL files with 42 sorry placeholders

Evidence

The audit took about 40 minutes. Here are the hard facts:

Source

The 11 Main Proof Files achieved a genuine PASS. All theorems elaborated successfully. Zero smuggled custom axioms.

Source

12 Files returned FAIL (42 placeholders found). These 12 files are located in ComparatorChallenges/. They are templates intentionally left with sorry for human contestants to fill out.

Source

Our sidecar enforces a four-valued verdict: PASS, FAIL, INVALID, NOT RUN.

Source