A Testing Strategy for Agent Patches: Property Checks, Fixtures, and a Flaky Freeze

Описание и применение

Три‑слойный шлюз (проверка свойств, замораживание фикстур, устранение нестабильных тестов) защищает от ошибок, скрытых обычными unit‑тестами, в патчах, генерируемых ИИ.

Задача

Agent-generated patches often pass unit tests but break production invariants, causing costly failures that unit tests miss.

Результат

Agent patches that pass the gate exhibit robust invariants, consistent data handling, and stable test outcomes, reducing production bugs.

Рабочий процесс

• 1. Run property-based tests (e.g., Hypothesis) to verify invariants across generated inputs.
• 2. Freeze and hash realistic fixtures; compare hashes in CI to detect unintended data changes.
• 3. Execute flaky freeze script to detect tests that fail inconsistently; quarantine flaky tests and require separate fixes.

Доказательства

Agent patches need a stronger contract. They need properties that hold across the whole input space.

Источник

Commit the output. Compare it in CI. If a fixture changes, the diff must be explicit and reviewed.

Источник