Spend a Free Tier on Repeats, Not Coverage

Description and use

A workflow that repeats a small frozen task pack on a free tier to assess model stability and report intervals.

Problem

Teams on a free tier lack tokens but can afford inference; they need a way to assess model stability without exhausting the allowance on breadth.

Outcome

The workflow consumes a free tier allowance efficiently by repeating a small, frozen task pack, yielding stability estimates and interval reporting instead of a single headline.

Workflow

• Freeze a small JSONL pack of tasks with exact or regex graders
• Hash each prompt to detect mid‑sweep edits
• Run the pack 5 times using a standard‑library Python harness
• Record tokens, wall‑clock time, and success/failure per attempt
• Export results to CSV for analysis

Evidence

The scarce resource on a free tier is not tokens; it is the number of independent looks you take at the same task.

Source

Five repeats on eight tasks yields forty observations and eight estimates of stability, while forty tasks run once yields forty observations and zero estimates of stability.

Source

The script below is a template, not a validated instrument; run it on two tasks first and inspect the rows before spending an allowance on it.

Source