48-Hour Field Notes: A Free Tier Is a Budget, Not a Benchmark

Description and use

Implemented spend tracking and citation validation in a CI pipeline using a free LLM tier to prevent unexpected failures from exceeding token limits.

Problem

A CI build failed on a branch untouched for nine days, revealing that the free tier of an LLM model was being exceeded without a clear way to measure or control spend.

Outcome

The workflow reduced unexpected build failures by enforcing a token budget and ensuring answers cite valid log lines, making expensive model usage rare and controlled.

Workflow

• 1. Create a spend ledger (ledger.py) that records token usage per day and blocks calls when the daily cap is exceeded.
• 2. Set the daily cap via TOKEN_CAP_PER_DAY environment variable (default 200000 tokens).
• 3. Implement an escalation ladder that defines stop conditions based on answer content and citation quality.
• 4. Add a floor check (floor.py) that verifies citations reference valid log lines.
• 5. Integrate the ledger, ladder, and floor check into the CI pipeline to guard against budget overruns and hallucinated answers.

Evidence

The first version of the ledger took forty lines and was worth more than every prompt tweak I made afterwards.

Source

Escalation is where most of the spend leaks out.

Source