Pin Property Seeds in CI Before You Score an Agent Patch

Description and use

Patches are scored against pinned seeds and a checked‑in corpus, removing flaky passes from random test generation.

Problem

Agent patches can pass CI by exploiting random seeds and shrinking examples, leading to false positives when tests are regenerated by models.

Outcome

Patches are evaluated against deterministic seeds and a fixed corpus, eliminating flaky passes caused by random test generation.

Workflow

• Confirm the agent diff does not modify oracle/
• Hash oracle/corpus.jsonl and compare to oracle/corpus.sha256
• Run the scoring suite with the pinned seed only
• Replay every corpus line against the patched tree
• Classify each line: still fails, now passes, or errored
• Apply the decision table (do not look at agent‑written test file for merge)

Evidence

Agent patches should be judged against pinned RNG seeds and a checked‑in corpus of shrunk examples, not against whatever Hypothesis or pytest‑randomly happens to explore on this CI worker.

Source

Keep independent oracles in a path the agent is not allowed to write. A CODEOWNERS rule or a CI path check is enough.

Source