A Routing Postmortem: When the Cheap Lane Stops Being Cheap

Description and use

A lightweight probe script now verifies the identity of the cheapest model before each batch cycle, preventing silent routing to expensive models and avoiding unexpected billing spikes.

Problem

A router that selects cheaper models silently started sending traffic to a more expensive model after a dependency update, causing unexpected billing spikes without affecting latency or error metrics.

Outcome

The added verification step prevents silent routing to expensive models, turning a potential billing surprise into an early, observable failure. The system now reliably routes batch jobs to the intended model group.

Workflow

• 1. Load routing configuration from routes.yaml.
• 2. Extract expected marker for the cheap lane.
• 3. Send a health check request to the cheap lane’s health_url with a timeout.
• 4. Verify the response status is 200; otherwise raise SystemExit and alert human.
• 5. Extract the actual marker from the response body.
• 6. Compare actual_marker to expected_marker; if mismatched, raise SystemExit and alert human.
• 7. If all checks pass, proceed with batch processing.

Evidence

We replaced the cost-sorted table with a rule that belongs to the batch job itself, not to the router. Each batch card now declares an expected model group, and the router must prove that the endpoint still reports that group before traffic moves.

Source

We also added a cheap-to-run verification step that runs before every batch cycle. It sends one tiny request to the cheapest configured path, captures a marker from the response, and compares that marker with the one stored alongside the model's configuration.

Source