A 10M-Token Budget Is a Test Plan: C++ Patch Trials on MonkeyCode's Free Tier

Description and use

A team uses a 10 M token budget to systematically test AI‑generated C++ patches on MonkeyCode’s free server, defining work units, pass criteria, and stop rules, then analyzing results by pass rate, tokens per task, and token distribution.

Problem

Teams often waste large free token allocations on unstructured AI coding experiments, leading to inefficient patch development and unclear success metrics.

Outcome

Teams can efficiently evaluate AI‑generated C++ patches within a fixed token budget, achieving clear pass rates and token usage insights while avoiding over‑generation and wasted prompts.

Workflow

• Define unit of work: minimal reproducer + expected behavior.
• Define pass condition: compile with -fsanitize=thread, run under TSan, exit within timeout.
• Define stop rule: spend at most 80% of the 10 M token budget.
• Create task list and reproducer files.
• Write budget_patch_trial.sh script to send prompts, compile patches, run TSan, and account tokens.
• Execute script, monitoring spent tokens and pass/fail counts.
• Analyze results: token distribution, pass rate, and harness reliability.
• Decide whether to stay on free tier based on matrix signals.
• Iterate: tighten prompts or max‑tokens if over‑generation occurs.

Evidence

The script below is the artifact. It reads a task list, sends each reproducer to the free server endpoint, compiles the returned patch, and runs it under TSan.

Source

Three numbers matter after a run: pass rate, tokens per task, and where the tokens went.

Source