Free Tokens, Free Server, and a Property Test That Refuses to Be Gamed

说明与用途

LLM 生成的补丁往往通过单元测试，但仍会破坏全局状态。通过在免费服务器上使用基于属性的测试 harness 并运行大量随机种子，开发者可以快速捕获单元测试遗漏的细微错误。

问题

LLM‑generated patches often pass unit tests but still corrupt global state, leading to failures in production.

结果

The property harness detected a subtle bug in a vector‑based ring buffer that unit tests missed, proving that randomised property tests on a free server are an effective, low‑cost verification strategy.

工作流

• Ask MonkeyCode’s free model to generate a patch
• Push the patch to a free server with a property‑test harness
• Run the harness with many random seeds and steps
• If the harness fails, review and fix the patch
• Repeat until the harness passes all seeds

证据

A property test with 1000 random seeds would have caught it at seed 17.

来源

The first time I ran this against an agent-generated ring buffer, the failure appeared at seed 17.

来源