The context‑per‑turn cost bomb: keep the cache warm, or run the loop in a cheaper model

说明与用途

代理循环每一步都会重新发送完整上下文，若缓存变冷会导致隐藏成本激增。解决方案包括保持循环在同一进程、避免早期上下文编辑、将简单子循环委托给更小模型，并监控缓存创建/读取指标。

问题

Agent loops re‑send the entire accumulated context on every turn, causing a hidden cost explosion when the prompt cache becomes cold.

结果

By keeping the cache warm and delegating high‑turn mechanical loops to cheaper models, the team reduced token costs by up to 10× for long agent loops and turned a silent monthly surprise into a two‑turn alert.

工作流

• 1. Ensure the agent loop runs within a single live process to avoid cache invalidation on restarts.
• 2. Append new context at the end; avoid rewriting the early context to keep the prefix stable.
• 3. For bounded, mechanical sub‑loops, delegate to a smaller, cheaper model with a tiny task‑scoped context.
• 4. Instrument the provider’s cache metrics (create vs read tokens) per turn and alert when creates dominate.
• 5. Review cache counters regularly to catch cold‑cache loops early.

证据

The cache keys on an unchanged prefix. Two things break that, and both are easy to do by accident: Restarting the process mid-loop… Editing early context mid-loop…

来源

Turn that into a cheap standing check: log the create‑versus‑read ratio per turn for any long‑running loop, and alert when a loop that should be warm is dominated by creates.

来源