Pin Property Seeds in CI Before You Score an Agent Patch

说明与用途

使用固定种子和已检查的语料库对补丁进行评分，消除随机测试生成导致的偶发通过。

问题

Agent patches can pass CI by exploiting random seeds and shrinking examples, leading to false positives when tests are regenerated by models.

结果

Patches are evaluated against deterministic seeds and a fixed corpus, eliminating flaky passes caused by random test generation.

工作流

• Confirm the agent diff does not modify oracle/
• Hash oracle/corpus.jsonl and compare to oracle/corpus.sha256
• Run the scoring suite with the pinned seed only
• Replay every corpus line against the patched tree
• Classify each line: still fails, now passes, or errored
• Apply the decision table (do not look at agent‑written test file for merge)

证据

Agent patches should be judged against pinned RNG seeds and a checked‑in corpus of shrunk examples, not against whatever Hypothesis or pytest‑randomly happens to explore on this CI worker.

来源

Keep independent oracles in a path the agent is not allowed to write. A CODEOWNERS rule or a CI path check is enough.

来源