Measure Your Free AI Tier Before You Trust It: Three Numbers in 20 Minutes

Description and use

A 60‑line probe script measures warm p95 latency, token accounting drift, and burst failure rate on free AI tiers.

Problem

Free AI tiers lack reliable performance and billing metrics, leading to uncertain latency, token accounting drift, and burst failure rates.

Outcome

The probe provides three actionable metrics—warm p95 latency, token accounting drift, and burst failure rate—allowing teams to validate free tier performance before scaling.

Workflow

• Set environment variables BASE_URL, MODEL, API_KEY, and PROMPT.
• Define call() to send a single /chat/completions request and record time and usage.
• Discard one cold-ish call, then perform 15 warm calls to collect latency data.
• Compute p95 latency from warm call durations.
• Sum reported total_tokens from warm responses and compare to local token count for accounting drift.
• Launch five concurrent requests to measure burst failure rate (429, 5xx, timeout).

Evidence

So I wrote a 60‑line probe. It runs in about twenty minutes on a free tier and it tells you three numbers nobody publishes.

Source

The three numbers

1. Warm p95 latency.
2. Token accounting drift.
3. Burst failure rate.

Source