A Free Tier Is a Contract Test, Not a Backend

Описание и применение

Бесплатные уровни моделей скрывают ошибки ограничения скорости, что приводит к сбоям в продакшене. Внедрение слою провайдера нормализует ошибки и повышает надёжность.

Задача

Free model tiers hide rate‑limit failures, causing production bugs when adapters are omitted.

Результат

Adopting a provider seam exposed hidden rate‑limit failures, preventing future 500 errors and improving reliability when using free model tiers.

Рабочий процесс

• Define a Failure type with kinds rate_limited, timeout, cancelled, upstream, invalid_output.
• Create a ModelProvider interface with a complete method returning Completion or Failure.
• Implement a MonkeyCode provider that checks config, makes a POST to /complete, handles 429 by returning a rate_limited Failure, and other non‑ok responses as upstream failures.
• Use the provider in route handlers, handling Failure values instead of exceptions.
• Run failure tests against the free tier to validate adapter resilience.
• Adjust retry logic based on retryAfterMs from rate_limited failures.

Доказательства

On staging it died at 03:14 with a 500 that was hiding a 429 inside three layers of try/catch

Источник

A free model tier is not a cheaper backend, it is the cheapest contract test you will ever be handed

Источник