Free Tokens, Free Server, and a Property Test That Refuses to Be Gamed

Описание и применение

Патчи, генерируемые LLM, часто проходят юнит‑тесты, но нарушают глобальное состояние. Используя бесплатный сервер и генератор случайных операций, можно быстро обнаружить ошибки, которые юнит‑тесты пропускают.

Задача

LLM‑generated patches often pass unit tests but still corrupt global state, leading to failures in production.

Результат

The property harness detected a subtle bug in a vector‑based ring buffer that unit tests missed, proving that randomised property tests on a free server are an effective, low‑cost verification strategy.

Рабочий процесс

• Ask MonkeyCode’s free model to generate a patch
• Push the patch to a free server with a property‑test harness
• Run the harness with many random seeds and steps
• If the harness fails, review and fix the patch
• Repeat until the harness passes all seeds

Доказательства

A property test with 1000 random seeds would have caught it at seed 17.

Источник

The first time I ran this against an agent-generated ring buffer, the failure appeared at seed 17.

Источник