A 10M-Token Budget Is a Test Plan: C++ Patch Trials on MonkeyCode's Free Tier

Описание и применение

Команда использует ограниченный бюджет токенов 10 М, чтобы систематически тестировать AI‑генерируемые патчи C++ на бесплатном сервере MonkeyCode, определяя единицы работы, критерии прохождения и правила остановки, а затем анализируя результаты по количеству пройденных задач, токенов на задачу и распределению токенов.

Задача

Teams often waste large free token allocations on unstructured AI coding experiments, leading to inefficient patch development and unclear success metrics.

Результат

Teams can efficiently evaluate AI‑generated C++ patches within a fixed token budget, achieving clear pass rates and token usage insights while avoiding over‑generation and wasted prompts.

Рабочий процесс

• Define unit of work: minimal reproducer + expected behavior.
• Define pass condition: compile with -fsanitize=thread, run under TSan, exit within timeout.
• Define stop rule: spend at most 80% of the 10 M token budget.
• Create task list and reproducer files.
• Write budget_patch_trial.sh script to send prompts, compile patches, run TSan, and account tokens.
• Execute script, monitoring spent tokens and pass/fail counts.
• Analyze results: token distribution, pass rate, and harness reliability.
• Decide whether to stay on free tier based on matrix signals.
• Iterate: tighten prompts or max‑tokens if over‑generation occurs.

Доказательства

The script below is the artifact. It reads a task list, sends each reproducer to the free server endpoint, compiles the returned patch, and runs it under TSan.

Источник

Three numbers matter after a run: pass rate, tokens per task, and where the tokens went.

Источник