Same-Prompt Green Is Not a Gate: Five Eval Myths

Description and use

Separating the oracle from the generator dispels the myth that a green test run means a finished gate, providing a reliable gate that truly verifies implementation against human‑defined expectations.

Problem

Teams treat a green test run from a model-generated test as a finished gate, but the tests may be authored by the model itself, leading to false confidence.

Outcome

Separating oracle from generator eliminates the myth that green tests equal a finished gate, providing a reliable gate that truly verifies implementation against human‑defined expectations.

Workflow

• Write contract (JSON cases) and failing tests first.
• Lock oracle directory so generator cannot modify it.
• Draft implementation in src/ with a free model; no tests are generated.
• Run scripts/run_oracle.sh on a free server or locally to execute oracle tests.
• Read the exit code; ignore chat summary as a gate.
• If tests fail, modify implementation only, not oracle or tests.
• If new requirements arise, edit oracle by hand and repeat the cycle.

Evidence

The tests passed, so the feature works.

Keep oracle/ out of the generate step. Keep src/ as the only writable target.

If it fails, change code. Not cases. Not tests.