48-Hour Field Notes: A Free Tier Is a Budget, Not a Benchmark

Описание и применение

Внедрён контроль расходов и проверка цитат для CI‑пайплайна, использующего бесплатный LLM‑тариф, чтобы предотвратить неожиданные сбои из‑за превышения лимита токенов.

Задача

A CI build failed on a branch untouched for nine days, revealing that the free tier of an LLM model was being exceeded without a clear way to measure or control spend.

Результат

The workflow reduced unexpected build failures by enforcing a token budget and ensuring answers cite valid log lines, making expensive model usage rare and controlled.

Рабочий процесс

• 1. Create a spend ledger (ledger.py) that records token usage per day and blocks calls when the daily cap is exceeded.
• 2. Set the daily cap via TOKEN_CAP_PER_DAY environment variable (default 200000 tokens).
• 3. Implement an escalation ladder that defines stop conditions based on answer content and citation quality.
• 4. Add a floor check (floor.py) that verifies citations reference valid log lines.
• 5. Integrate the ledger, ladder, and floor check into the CI pipeline to guard against budget overruns and hallucinated answers.

Доказательства

The first version of the ledger took forty lines and was worth more than every prompt tweak I made afterwards.

Источник

Escalation is where most of the spend leaks out.

Источник