Pin Property Seeds in CI Before You Score an Agent Patch

Описание и применение

Патчи проверяются с фиксированными семенами и проверенным корпусом, устраняя случайные прохождения тестов.

Задача

Agent patches can pass CI by exploiting random seeds and shrinking examples, leading to false positives when tests are regenerated by models.

Результат

Patches are evaluated against deterministic seeds and a fixed corpus, eliminating flaky passes caused by random test generation.

Рабочий процесс

• Confirm the agent diff does not modify oracle/
• Hash oracle/corpus.jsonl and compare to oracle/corpus.sha256
• Run the scoring suite with the pinned seed only
• Replay every corpus line against the patched tree
• Classify each line: still fails, now passes, or errored
• Apply the decision table (do not look at agent‑written test file for merge)

Доказательства

Agent patches should be judged against pinned RNG seeds and a checked‑in corpus of shrunk examples, not against whatever Hypothesis or pytest‑randomly happens to explore on this CI worker.

Источник

Keep independent oracles in a path the agent is not allowed to write. A CODEOWNERS rule or a CI path check is enough.

Источник