The context‑per‑turn cost bomb: keep the cache warm, or run the loop in a cheaper model

Описание и применение

В агентских циклах каждый ход пересылает весь контекст, что приводит к скрытому росту стоимости, если кэш становится холодным. Решения: держать цикл в одном процессе, не редактировать ранний контекст, делегировать простые подциклы небольшим моделям и отслеживать метрики кэша.

Задача

Agent loops re‑send the entire accumulated context on every turn, causing a hidden cost explosion when the prompt cache becomes cold.

Результат

By keeping the cache warm and delegating high‑turn mechanical loops to cheaper models, the team reduced token costs by up to 10× for long agent loops and turned a silent monthly surprise into a two‑turn alert.

Рабочий процесс

• 1. Ensure the agent loop runs within a single live process to avoid cache invalidation on restarts.
• 2. Append new context at the end; avoid rewriting the early context to keep the prefix stable.
• 3. For bounded, mechanical sub‑loops, delegate to a smaller, cheaper model with a tiny task‑scoped context.
• 4. Instrument the provider’s cache metrics (create vs read tokens) per turn and alert when creates dominate.
• 5. Review cache counters regularly to catch cold‑cache loops early.

Доказательства

The cache keys on an unchanged prefix. Two things break that, and both are easy to do by accident: Restarting the process mid-loop… Editing early context mid-loop…

Источник

Turn that into a cheap standing check: log the create‑versus‑read ratio per turn for any long‑running loop, and alert when a loop that should be warm is dominated by creates.

Источник