A 10M-Token Budget Is a Test Plan: C++ Patch Trials on MonkeyCode's Free Tier

说明与用途

团队使用10M令牌预算，在MonkeyCode免费服务器上系统地测试AI生成的C++补丁，定义工作单元、通过标准和停止规则，然后按通过率、每任务令牌数和令牌分布分析结果。

问题

Teams often waste large free token allocations on unstructured AI coding experiments, leading to inefficient patch development and unclear success metrics.

结果

Teams can efficiently evaluate AI‑generated C++ patches within a fixed token budget, achieving clear pass rates and token usage insights while avoiding over‑generation and wasted prompts.

工作流

• Define unit of work: minimal reproducer + expected behavior.
• Define pass condition: compile with -fsanitize=thread, run under TSan, exit within timeout.
• Define stop rule: spend at most 80% of the 10 M token budget.
• Create task list and reproducer files.
• Write budget_patch_trial.sh script to send prompts, compile patches, run TSan, and account tokens.
• Execute script, monitoring spent tokens and pass/fail counts.
• Analyze results: token distribution, pass rate, and harness reliability.
• Decide whether to stay on free tier based on matrix signals.
• Iterate: tighten prompts or max‑tokens if over‑generation occurs.

证据

The script below is the artifact. It reads a task list, sends each reproducer to the free server endpoint, compiles the returned patch, and runs it under TSan.

来源

Three numbers matter after a run: pass rate, tokens per task, and where the tokens went.

来源