Build a Tiny Token Meter Before You Trust a Free Model Tier

Описание и применение

Небольшой HTTP‑прокси, записывающий токены каждого запроса к бесплатному AI‑эндпоинту, предотвращает непреднамеренное исчерпание квоты и сохраняет логи в JSONL.

Задача

Free model tiers can be exhausted silently by retry loops that ignore usage data, leading to unexpected quota depletion before a demo or usage dashboard is checked.

Результат

The token meter prevented accidental quota exhaustion, provided per‑request token accounting, and logged usage for auditability.

Рабочий процесс

• Configure environment variables: UPSTREAM_URL, UPSTREAM_TOKEN, DAY_TOKEN_CAP, LEDGER_PATH, PORT, METER_STUB.
• Run meter.py as a local HTTP proxy on the specified PORT.
• Client sends requests to the proxy instead of directly to the upstream endpoint.
• Proxy forwards the request, receives the response, extracts prompt and completion tokens from the usage block.
• Proxy writes a JSON line to the ledger with day, timestamp, prompt_tokens, completion_tokens, and total_tokens.
• Proxy checks the accumulated tokens for the day; if the sum exceeds DAY_TOKEN_CAP, it returns an error and refuses new calls.
• Optional stub mode (METER_STUB=1) returns a deterministic response for testing without consuming real tokens.

Доказательства

The meter should do three things and nothing else. Count tokens per request from the upstream's own usage block, append one JSON line per call to a ledger, and refuse new calls once today's total crosses a hard cap.

Источник

One file, meter.py. It is an HTTP proxy: you point your client at it, it forwards to the upstream, it records what came back.

Источник

The upstream call has a stub branch, and that branch is the whole reason this article is testable at 11:40pm without spending anything.

Источник