Review Budget, Not Token Budget: A Pairing Session on Free-Tier Model Access

Description and use

The team bounded free-tier model usage to a single directory and matched AI patch review effort to human capacity to avoid overcommitment.

Problem

Teams often misjudge free-tier AI model limits by focusing on token counts instead of human review capacity, leading to overcommitment and wasted effort.

Outcome

The team limited free-tier usage to a single directory and matched AI patch review effort to human capacity, preventing overcommitment and ensuring quality.

Workflow

• 1. Reframe the question from “which model is best” to “what task to automate”.
• 2. Estimate token usage with a custom script (review_budget.py).
• 3. Run the estimator on the migration branch to get token estimates.
• 4. Measure review effort: count patches and estimate minutes per patch.
• 5. Compare total review minutes to available reviewer capacity.
• 6. Replay three historical patches to validate model reliability.
• 7. Apply a rule: only allow the free tier to touch a single bounded directory.
• 8. Execute the bounded slice, generating patches within review capacity.
• 9. Review AI patches with the same process as human patches.

Evidence

425,000 tokens against a 10 million allowance. The quota was not the constraint.

Source

46 draft pull requests. Careful review of each patch took about 15 minutes… 11.5 hours of review attention.

Source