Free Compute Is a Falsification Lab, Not a Backend

Description and use

Using free token lanes as production leads to undetected contract drift. A simple probe compares two lanes’ responses, detecting differences in status, latency, and schema.

Problem

Treating free token lanes as production leads to undetected contract drift and failures.

Outcome

The probe revealed that the free lane returned a 200 status with a different response shape, causing a timeout budget blow and mis‑parsed summaries.

Workflow

• Define a JSONL of test cases with expected constraints
• Set environment variables for lane name, base URL, API key, model, and budget
• Load cases from golden/tickets.jsonl
• For each case, send POST to /chat/completions with model, temperature 0, and messages
• Record status code, latency, and whether payload contains expected schema
• Append results to a ledger
• Run probe twice for each lane and diff ledgers
• Identify drift in status, latency, or schema

Evidence

Last Tuesday a support copilot I maintain wrote "Sure! Here’s the summary:" into a customer's issue field, and that sentence sat there for eleven minutes before a human noticed it.

Source

A second lane does not usually fail by returning errors; it fails by returning success with a different shape.

Source

The artifact I want you to steal is a golden‑set probe that runs the identical request corpus against two lanes and writes two ledgers you can diff.

Source