Hardening Claude Code at Scale: Surviving Context Bloat, Rate‑Limit Cascades, and Prompt Cache Eviction

Описание и применение

Внедрение посреднической архитектуры с Envoy и LiteLLM устраняет рост контекста, падение кэша и синхронные потоки 429, позволяя безопасно использовать anthropics/claude‑code в корпоративных рабочих нагрузках.

Задача

Automated monorepo migrations using anthropics/claude‑code caused runaway token spend and upstream 429 locks due to uncontrolled context bloat, cache thrashing, and synchronized retry storms.

Результат

The mediation architecture reduced token spend, restored cache hit ratios above 85%, and eliminated synchronized 429 retry storms, allowing continuous automated repo sweeps without upstream lockouts.

Рабочий процесс

• 1. Deploy Envoy sidecar listening on 127.0.0.1:8080 with strict circuit breakers and retry policy.
• 2. Configure Envoy to forward requests to a LiteLLM mediator running on 127.0.0.1:4000.
• 3. Set LiteLLM to pin prompt prefixes, enforce token budget clamps, and route to enterprise gateway (https://api.b‑lost.com/v1).
• 4. Run anthropics/claude‑code CLI with ANTHROPIC_BASE_URL pointing to the local Envoy proxy.
• 5. Monitor token usage, cache hit ratios, and rate‑limit metrics to confirm stability.

Доказательства

our background agent cluster incinerated $940 of API credit in twenty‑two minutes before triggering a hard upstream 429 lock

Источник

prompt context expands monotonically. Without an intermediate mediation layer, every minor retry resends a 140k-token payload over cold connections, evicting upstream prompt caches and stalling the entire development pipeline.

Источник

We decoupled client CLI instances from upstream model providers by routing all CLI traffic through an edge mediation topology

Источник