Hardening Claude Code at Scale: Surviving Context Bloat, Rate‑Limit Cascades, and Prompt Cache Eviction

Description and use

Deploying an Envoy sidecar and LiteLLM mediator eliminates context bloat, cache thrashing, and synchronized 429 storms, enabling reliable use of anthropics/claude‑code in enterprise pipelines.

Problem

Automated monorepo migrations using anthropics/claude‑code caused runaway token spend and upstream 429 locks due to uncontrolled context bloat, cache thrashing, and synchronized retry storms.

Outcome

The mediation architecture reduced token spend, restored cache hit ratios above 85%, and eliminated synchronized 429 retry storms, allowing continuous automated repo sweeps without upstream lockouts.

Workflow

• 1. Deploy Envoy sidecar listening on 127.0.0.1:8080 with strict circuit breakers and retry policy.
• 2. Configure Envoy to forward requests to a LiteLLM mediator running on 127.0.0.1:4000.
• 3. Set LiteLLM to pin prompt prefixes, enforce token budget clamps, and route to enterprise gateway (https://api.b‑lost.com/v1).
• 4. Run anthropics/claude‑code CLI with ANTHROPIC_BASE_URL pointing to the local Envoy proxy.
• 5. Monitor token usage, cache hit ratios, and rate‑limit metrics to confirm stability.

Evidence

our background agent cluster incinerated $940 of API credit in twenty‑two minutes before triggering a hard upstream 429 lock

Source

prompt context expands monotonically. Without an intermediate mediation layer, every minor retry resends a 140k-token payload over cold connections, evicting upstream prompt caches and stalling the entire development pipeline.

Source

We decoupled client CLI instances from upstream model providers by routing all CLI traffic through an edge mediation topology

Source