A Routing Postmortem: When the Cheap Lane Stops Being Cheap

Описание и применение

Добавлен проверочный скрипт, который перед каждым пакетным циклом подтверждает, что «дешёвая» модель действительно отвечает, предотвращая незаметное переключение на дорогую модель и неожиданные расходы.

Задача

A router that selects cheaper models silently started sending traffic to a more expensive model after a dependency update, causing unexpected billing spikes without affecting latency or error metrics.

Результат

The added verification step prevents silent routing to expensive models, turning a potential billing surprise into an early, observable failure. The system now reliably routes batch jobs to the intended model group.

Рабочий процесс

• 1. Load routing configuration from routes.yaml.
• 2. Extract expected marker for the cheap lane.
• 3. Send a health check request to the cheap lane’s health_url with a timeout.
• 4. Verify the response status is 200; otherwise raise SystemExit and alert human.
• 5. Extract the actual marker from the response body.
• 6. Compare actual_marker to expected_marker; if mismatched, raise SystemExit and alert human.
• 7. If all checks pass, proceed with batch processing.

Доказательства

We replaced the cost-sorted table with a rule that belongs to the batch job itself, not to the router. Each batch card now declares an expected model group, and the router must prove that the endpoint still reports that group before traffic moves.

Источник

We also added a cheap-to-run verification step that runs before every batch cycle. It sends one tiny request to the cheapest configured path, captures a marker from the response, and compares that marker with the one stored alongside the model's configuration.

Источник