The context‑per‑turn cost bomb: keep the cache warm, or run the loop in a cheaper model

Description and use

Agent loops resend the entire context each turn, causing hidden cost spikes when the cache goes cold. Fixes include keeping the loop in one process, avoiding early context edits, delegating simple sub‑loops to cheaper models, and monitoring cache create/read metrics.

Problem

Agent loops re‑send the entire accumulated context on every turn, causing a hidden cost explosion when the prompt cache becomes cold.

Outcome

By keeping the cache warm and delegating high‑turn mechanical loops to cheaper models, the team reduced token costs by up to 10× for long agent loops and turned a silent monthly surprise into a two‑turn alert.

Workflow

• 1. Ensure the agent loop runs within a single live process to avoid cache invalidation on restarts.
• 2. Append new context at the end; avoid rewriting the early context to keep the prefix stable.
• 3. For bounded, mechanical sub‑loops, delegate to a smaller, cheaper model with a tiny task‑scoped context.
• 4. Instrument the provider’s cache metrics (create vs read tokens) per turn and alert when creates dominate.
• 5. Review cache counters regularly to catch cold‑cache loops early.

Evidence

The cache keys on an unchanged prefix. Two things break that, and both are easy to do by accident: Restarting the process mid-loop… Editing early context mid-loop…

Source

Turn that into a cheap standing check: log the create‑versus‑read ratio per turn for any long‑running loop, and alert when a loop that should be warm is dominated by creates.

Source