Properties, Fixtures, and a Flaky Freeze: A Three-Layer Gate for Agent Patches

Описание и применение

Три‑слойный шлюз с проверками свойств, фикстурами и «неустойчивой заморозкой» повышает надёжность патчей, созданных агентами, предотвращая переобучение, ошибки на границах и недетерминированность.

Задача

Agent patches often pass tests that only cover the prompt examples, leading to overfitting, edge hallucination, and nondeterminism.

Результат

The gate catches edge‑case failures and ensures patches generalize beyond the prompt examples, improving merge confidence.

Рабочий процесс

• 1. Define property checks that generate random inputs and assert invariants.
• 2. Implement a minimal property harness with a fixed‑seed RNG for reproducibility.
• 3. Write property functions (e.g., round‑trip and additive for parse_duration).
• 4. Run the harness with a fixed number of trials (e.g., 2000).
• 5. If any trial fails, report the failing input and exit with non‑zero status.
• 6. Use the flaky freeze to ensure the gate signal is clean before merging.

Доказательства

Property checks, fixtures, and a flaky freeze. Each layer answers a different question.

Источник

The fixed seed is deliberate. A gate that fails on trial 731 today and trial 402 tomorrow is useless for merge decisions.

Источник