A Free Tier Is a Contract Test, Not a Backend

说明与用途

免费模型层隐藏速率限制错误，导致生产故障。实现提供者层可规范错误并提高可靠性。

问题

Free model tiers hide rate‑limit failures, causing production bugs when adapters are omitted.

结果

Adopting a provider seam exposed hidden rate‑limit failures, preventing future 500 errors and improving reliability when using free model tiers.

工作流

• Define a Failure type with kinds rate_limited, timeout, cancelled, upstream, invalid_output.
• Create a ModelProvider interface with a complete method returning Completion or Failure.
• Implement a MonkeyCode provider that checks config, makes a POST to /complete, handles 429 by returning a rate_limited Failure, and other non‑ok responses as upstream failures.
• Use the provider in route handlers, handling Failure values instead of exceptions.
• Run failure tests against the free tier to validate adapter resilience.
• Adjust retry logic based on retryAfterMs from rate_limited failures.

证据

On staging it died at 03:14 with a 500 that was hiding a 429 inside three layers of try/catch

来源

A free model tier is not a cheaper backend, it is the cheapest contract test you will ever be handed

来源