Spend a Free Tier on Repeats, Not Coverage

Описание и применение

Технологический процесс повторения небольшого пакета задач на бесплатном тарифе позволяет оценить стабильность модели, используя ограниченные ресурсы.

Задача

Teams on a free tier lack tokens but can afford inference; they need a way to assess model stability without exhausting the allowance on breadth.

Результат

The workflow consumes a free tier allowance efficiently by repeating a small, frozen task pack, yielding stability estimates and interval reporting instead of a single headline.

Рабочий процесс

• Freeze a small JSONL pack of tasks with exact or regex graders
• Hash each prompt to detect mid‑sweep edits
• Run the pack 5 times using a standard‑library Python harness
• Record tokens, wall‑clock time, and success/failure per attempt
• Export results to CSV for analysis

Доказательства

The scarce resource on a free tier is not tokens; it is the number of independent looks you take at the same task.

Источник

Five repeats on eight tasks yields forty observations and eight estimates of stability, while forty tasks run once yields forty observations and zero estimates of stability.

Источник

The script below is a template, not a validated instrument; run it on two tasks first and inspect the rows before spending an allowance on it.

Источник