Same-Prompt Green Is Not a Gate: Five Eval Myths

说明与用途

将oracle与生成器分离，消除了绿色测试运行即完成门的误解，提供了真正根据人工定义的期望验证实现的可靠门。

问题

Teams treat a green test run from a model-generated test as a finished gate, but the tests may be authored by the model itself, leading to false confidence.

结果

Separating oracle from generator eliminates the myth that green tests equal a finished gate, providing a reliable gate that truly verifies implementation against human‑defined expectations.

工作流

• Write contract (JSON cases) and failing tests first.
• Lock oracle directory so generator cannot modify it.
• Draft implementation in src/ with a free model; no tests are generated.
• Run scripts/run_oracle.sh on a free server or locally to execute oracle tests.
• Read the exit code; ignore chat summary as a gate.
• If tests fail, modify implementation only, not oracle or tests.
• If new requirements arise, edit oracle by hand and repeat the cycle.

证据

The tests passed, so the feature works.

Keep oracle/ out of the generate step. Keep src/ as the only writable target.

If it fails, change code. Not cases. Not tests.