A Routing Postmortem: When the Cheap Lane Stops Being Cheap

说明与用途

在每个批处理周期前执行轻量级探测脚本，验证“廉价”模型的身份，防止无声切换到昂贵模型并避免意外费用。

问题

A router that selects cheaper models silently started sending traffic to a more expensive model after a dependency update, causing unexpected billing spikes without affecting latency or error metrics.

结果

The added verification step prevents silent routing to expensive models, turning a potential billing surprise into an early, observable failure. The system now reliably routes batch jobs to the intended model group.

工作流

• 1. Load routing configuration from routes.yaml.
• 2. Extract expected marker for the cheap lane.
• 3. Send a health check request to the cheap lane’s health_url with a timeout.
• 4. Verify the response status is 200; otherwise raise SystemExit and alert human.
• 5. Extract the actual marker from the response body.
• 6. Compare actual_marker to expected_marker; if mismatched, raise SystemExit and alert human.
• 7. If all checks pass, proceed with batch processing.

证据

We replaced the cost-sorted table with a rule that belongs to the batch job itself, not to the router. Each batch card now declares an expected model group, and the router must prove that the endpoint still reports that group before traffic moves.

来源

We also added a cheap-to-run verification step that runs before every batch cycle. It sends one tiny request to the cheapest configured path, captures a marker from the response, and compares that marker with the one stored alongside the model's configuration.

来源