Hardening Claude Code at Scale: Surviving Context Bloat, Rate‑Limit Cascades, and Prompt Cache Eviction

说明与用途

通过部署 Envoy 侧车和 LiteLLM 中介，消除了上下文膨胀、缓存失效和同步 429 风暴，使企业级工作流能够可靠使用 anthropics/claude‑code。

问题

Automated monorepo migrations using anthropics/claude‑code caused runaway token spend and upstream 429 locks due to uncontrolled context bloat, cache thrashing, and synchronized retry storms.

结果

The mediation architecture reduced token spend, restored cache hit ratios above 85%, and eliminated synchronized 429 retry storms, allowing continuous automated repo sweeps without upstream lockouts.

工作流

• 1. Deploy Envoy sidecar listening on 127.0.0.1:8080 with strict circuit breakers and retry policy.
• 2. Configure Envoy to forward requests to a LiteLLM mediator running on 127.0.0.1:4000.
• 3. Set LiteLLM to pin prompt prefixes, enforce token budget clamps, and route to enterprise gateway (https://api.b‑lost.com/v1).
• 4. Run anthropics/claude‑code CLI with ANTHROPIC_BASE_URL pointing to the local Envoy proxy.
• 5. Monitor token usage, cache hit ratios, and rate‑limit metrics to confirm stability.

证据

our background agent cluster incinerated $940 of API credit in twenty‑two minutes before triggering a hard upstream 429 lock

来源

prompt context expands monotonically. Without an intermediate mediation layer, every minor retry resends a 140k-token payload over cold connections, evicting upstream prompt caches and stalling the entire development pipeline.

来源

We decoupled client CLI instances from upstream model providers by routing all CLI traffic through an edge mediation topology

来源