Free Compute Is a Falsification Lab, Not a Backend

说明与用途

将免费令牌层视为生产环境会导致合同漂移未被检测。提出一个简单的探针比较两条线路的响应，检测状态、延迟和模式的差异。

问题

Treating free token lanes as production leads to undetected contract drift and failures.

结果

The probe revealed that the free lane returned a 200 status with a different response shape, causing a timeout budget blow and mis‑parsed summaries.

工作流

• Define a JSONL of test cases with expected constraints
• Set environment variables for lane name, base URL, API key, model, and budget
• Load cases from golden/tickets.jsonl
• For each case, send POST to /chat/completions with model, temperature 0, and messages
• Record status code, latency, and whether payload contains expected schema
• Append results to a ledger
• Run probe twice for each lane and diff ledgers
• Identify drift in status, latency, or schema

证据

Last Tuesday a support copilot I maintain wrote "Sure! Here’s the summary:" into a customer's issue field, and that sentence sat there for eleven minutes before a human noticed it.

来源

A second lane does not usually fail by returning errors; it fails by returning success with a different shape.

来源

The artifact I want you to steal is a golden‑set probe that runs the identical request corpus against two lanes and writes two ledgers you can diff.

来源