48-Hour Field Notes: A Free Tier Is a Budget, Not a Benchmark

说明与用途

在使用免费 LLM 计费的 CI 流水线中实现了支出跟踪和引用验证，以防止因超出令牌限制而导致的意外失败。

问题

A CI build failed on a branch untouched for nine days, revealing that the free tier of an LLM model was being exceeded without a clear way to measure or control spend.

结果

The workflow reduced unexpected build failures by enforcing a token budget and ensuring answers cite valid log lines, making expensive model usage rare and controlled.

工作流

• 1. Create a spend ledger (ledger.py) that records token usage per day and blocks calls when the daily cap is exceeded.
• 2. Set the daily cap via TOKEN_CAP_PER_DAY environment variable (default 200000 tokens).
• 3. Implement an escalation ladder that defines stop conditions based on answer content and citation quality.
• 4. Add a floor check (floor.py) that verifies citations reference valid log lines.
• 5. Integrate the ledger, ladder, and floor check into the CI pipeline to guard against budget overruns and hallucinated answers.

证据

The first version of the ledger took forty lines and was worth more than every prompt tweak I made afterwards.

来源

Escalation is where most of the spend leaks out.

来源