Benchmark a Free AI Coding Tier on a Cold Server

Описание и применение

Пошаговый процесс измерения реальной задержки и надёжности бесплатных AI‑тарифов на холодном сервере, включая набор тестов, сбор метрик и публикацию результатов.

Задача

Developers need realistic latency and reliability metrics for free AI coding tiers, but warm local runs hide first‑token latency and variance.

Результат

The workflow produces reproducible cold‑start latency and reliability metrics, exposing hidden costs and failure modes that marketing screenshots omit.

Рабочий процесс

• 1. Prepare a contract‑based dataset with deterministic, bug‑repair, and repository‑context prompts, each with an explicit pass/fail check.
• 2. Store the dataset in a Python list of dicts (e.g., CASES).
• 3. Write a harness that sends each prompt to the API endpoint, records start time, captures the timestamp of the first streamed token, and records total completion time.
• 4. Reset the conversation before every call to avoid context leakage.
• 5. Run each case three times with the same seed, temperature, and prompt order to collect repeatable samples.
• 6. Compute median and p95 for first‑token latency, total time, and pass rate; flag timeouts, empty responses, and rate‑limit errors.
• 7. Calculate token consumption and cost per task relative to the free allowance.
• 8. Record server region, date range, and session age to contextualize results.
• 9. Publish the dataset, raw timestamps, and failure distribution for transparency.
• 10. Iterate the harness to add retries, rate‑limit handling, and stream parsing for production use.

Доказательства

Benchmarking a free AI coding tier on a cold remote server produces a number that marketing screenshots rarely show.

Источник

The harness sends each prompt to an API endpoint and records two timestamps. The difference between the first streamed token and the stop event yields the two numbers that matter, first-token latency and total completion time.

Источник