Same-Prompt Green Is Not a Gate: Five Eval Myths

Описание и применение

Разделение оркестра и генератора устраняет миф о том, что зеленый тестовый запуск означает завершённый шлюз, обеспечивая надёжную проверку реализации по человеческим ожиданиям.

Задача

Teams treat a green test run from a model-generated test as a finished gate, but the tests may be authored by the model itself, leading to false confidence.

Результат

Separating oracle from generator eliminates the myth that green tests equal a finished gate, providing a reliable gate that truly verifies implementation against human‑defined expectations.

Рабочий процесс

• Write contract (JSON cases) and failing tests first.
• Lock oracle directory so generator cannot modify it.
• Draft implementation in src/ with a free model; no tests are generated.
• Run scripts/run_oracle.sh on a free server or locally to execute oracle tests.
• Read the exit code; ignore chat summary as a gate.
• If tests fail, modify implementation only, not oracle or tests.
• If new requirements arise, edit oracle by hand and repeat the cycle.

Доказательства

The tests passed, so the feature works.

Keep oracle/ out of the generate step. Keep src/ as the only writable target.

If it fails, change code. Not cases. Not tests.