Properties, Fixtures, and a Flaky Freeze: A Three-Layer Gate for Agent Patches

Description and use

A three‑layer gate—property checks, fixtures, and a flaky freeze—improves the reliability of agent‑written patches by preventing overfitting, edge hallucination, and nondeterminism.

Problem

Agent patches often pass tests that only cover the prompt examples, leading to overfitting, edge hallucination, and nondeterminism.

Outcome

The gate catches edge‑case failures and ensures patches generalize beyond the prompt examples, improving merge confidence.

Workflow

• 1. Define property checks that generate random inputs and assert invariants.
• 2. Implement a minimal property harness with a fixed‑seed RNG for reproducibility.
• 3. Write property functions (e.g., round‑trip and additive for parse_duration).
• 4. Run the harness with a fixed number of trials (e.g., 2000).
• 5. If any trial fails, report the failing input and exit with non‑zero status.
• 6. Use the flaky freeze to ensure the gate signal is clean before merging.

Evidence

Property checks, fixtures, and a flaky freeze. Each layer answers a different question.

Source

The fixed seed is deliberate. A gate that fails on trial 731 today and trial 402 tomorrow is useless for merge decisions.

Source