Quarantine Agent Patches on a Free Ephemeral Server

说明与用途

在干净的临时服务器上验证 AI 生成的补丁可以发现本地测试未捕获的隐藏错误，降低发布错误代码的风险。

问题

AI‑generated patches often pass local tests but fail in a different environment due to cached state or hidden assumptions.

结果

The quarantine workflow catches hidden bugs (e.g., timezone issues) that local tests miss, reducing the risk of shipping faulty patches.

工作流

• Create a branch with the agent’s patch and open a PR or equivalent
• Start a free MonkeyCode server and obtain a shell
• Clone the repository on the server using a shallow clone (git clone --depth 1)
• Check out the patch branch
• Install the project in editable mode (pip install -e .)
• Run focused tests (e.g., property‑based tests for the patched files) with pytest -q
• If a test fails, analyze the traceback and optionally ask an LLM for hypotheses
• Use fixtures to pin environment variables (e.g., timezone) to avoid state leakage
• Mark flaky tests with @pytest.mark.skipif and run them manually when ready
• Wrap the steps in a quarantine.sh script that SSHs into the server and executes the commands
• Repeat for each suspicious patch

证据

I now do it for every suspicious one. It caught two real bugs in the last week. Both were timezone issues.

来源