A Testing Strategy for Agent Patches: Property Checks, Fixtures, and a Flaky Freeze

Description and use

A three‑layer gate—property checks, frozen fixtures, and flaky freeze—protects against bugs hidden by conventional unit tests in AI‑generated patches.

Problem

Agent-generated patches often pass unit tests but break production invariants, causing costly failures that unit tests miss.

Outcome

Agent patches that pass the gate exhibit robust invariants, consistent data handling, and stable test outcomes, reducing production bugs.

Workflow

• 1. Run property-based tests (e.g., Hypothesis) to verify invariants across generated inputs.
• 2. Freeze and hash realistic fixtures; compare hashes in CI to detect unintended data changes.
• 3. Execute flaky freeze script to detect tests that fail inconsistently; quarantine flaky tests and require separate fixes.

Evidence

Agent patches need a stronger contract. They need properties that hold across the whole input space.

Source

Commit the output. Compare it in CI. If a fixture changes, the diff must be explicit and reviewed.

Source