A Free Tier Is a Contract Test, Not a Backend

Description and use

Free model tiers hide rate‑limit errors, causing production failures. Implementing a provider seam normalizes failures and improves reliability.

Problem

Free model tiers hide rate‑limit failures, causing production bugs when adapters are omitted.

Outcome

Adopting a provider seam exposed hidden rate‑limit failures, preventing future 500 errors and improving reliability when using free model tiers.

Workflow

• Define a Failure type with kinds rate_limited, timeout, cancelled, upstream, invalid_output.
• Create a ModelProvider interface with a complete method returning Completion or Failure.
• Implement a MonkeyCode provider that checks config, makes a POST to /complete, handles 429 by returning a rate_limited Failure, and other non‑ok responses as upstream failures.
• Use the provider in route handlers, handling Failure values instead of exceptions.
• Run failure tests against the free tier to validate adapter resilience.
• Adjust retry logic based on retryAfterMs from rate_limited failures.

Evidence

On staging it died at 03:14 with a 500 that was hiding a 429 inside three layers of try/catch

Source

A free model tier is not a cheaper backend, it is the cheapest contract test you will ever be handed

Source