Free Compute Is a Falsification Lab, Not a Backend

Описание и применение

Использование бесплатных токенов как продакшн‑слоя приводит к незамеченному сдвигу контрактов. Предложен простой «проб» для сравнения ответов двух слоёв, выявляющий различия в статусе, задержке и схеме.

Задача

Treating free token lanes as production leads to undetected contract drift and failures.

Результат

The probe revealed that the free lane returned a 200 status with a different response shape, causing a timeout budget blow and mis‑parsed summaries.

Рабочий процесс

• Define a JSONL of test cases with expected constraints
• Set environment variables for lane name, base URL, API key, model, and budget
• Load cases from golden/tickets.jsonl
• For each case, send POST to /chat/completions with model, temperature 0, and messages
• Record status code, latency, and whether payload contains expected schema
• Append results to a ledger
• Run probe twice for each lane and diff ledgers
• Identify drift in status, latency, or schema

Доказательства

Last Tuesday a support copilot I maintain wrote "Sure! Here’s the summary:" into a customer's issue field, and that sentence sat there for eleven minutes before a human noticed it.

Источник

A second lane does not usually fail by returning errors; it fails by returning success with a different shape.

Источник

The artifact I want you to steal is a golden‑set probe that runs the identical request corpus against two lanes and writes two ledgers you can diff.

Источник