Spend a Free Tier on Repeats, Not Coverage

说明与用途

在免费套餐上重复小型冻结任务包的工作流程，以评估模型稳定性并报告区间。

问题

Teams on a free tier lack tokens but can afford inference; they need a way to assess model stability without exhausting the allowance on breadth.

结果

The workflow consumes a free tier allowance efficiently by repeating a small, frozen task pack, yielding stability estimates and interval reporting instead of a single headline.

工作流

• Freeze a small JSONL pack of tasks with exact or regex graders
• Hash each prompt to detect mid‑sweep edits
• Run the pack 5 times using a standard‑library Python harness
• Record tokens, wall‑clock time, and success/failure per attempt
• Export results to CSV for analysis

证据

The scarce resource on a free tier is not tokens; it is the number of independent looks you take at the same task.

来源

Five repeats on eight tasks yields forty observations and eight estimates of stability, while forty tasks run once yields forty observations and zero estimates of stability.

来源

The script below is a template, not a validated instrument; run it on two tasks first and inspect the rows before spending an allowance on it.

来源