Freeze the Counterexample, Not the Test File

Description and use

Ensures fixture immutability, preventing agent tampering with tests.

Problem

Agent patches often sabotage tests by rewriting golden files, wrapping assertions, or inflating timeouts, making it hard to detect genuine failures.

Outcome

Tests now run against immutable, verified fixtures, eliminating hidden failures caused by agent patches.

Workflow

• 1. Store fixture bytes in oracles/objects/<sha256> and map names in oracles/manifest.json.
• 2. Run tools/verify_manifest.py before tests to ensure all digests match stored objects and no drift exists.
• 3. Use property suite to load fixtures by digest, not path.
• 4. Record any new counterexamples in the flake ledger and add them as objects.
• 5. Fail CI if manifest names change or objects drift, but allow orphans as warnings.

Evidence

Phase-1 CI should fail closed if a name moved or a file drifted.

Source