I tried to build a "token optimization stack" for coding agents. Here’s why I killed it.

Описание и применение

Автор попытался создать набор инструментов для снижения затрат на токены при работе с Claude Code, но пилот показал, что экономия токенов маскирует ошибки в работе агента, а расходы на полный тест превысили бюджет.

Задача

Token limits and high costs when using Claude Code for engineering tasks.

Результат

The token-optimization stack produced misleading savings figures; a 97% token reduction corresponded to zero solved tasks. The pilot cost $5.60 per run and projected $1,200–$2,400 for full testing, with correctness failures on weak models.

Рабочий процесс

• 1. Identify token-heavy areas: context reading, re-reading, verbose output, codebase navigation.
• 2. Assemble a stack of five tools: Graphify, Serena, Headroom, LiteLLM, Caveman.
• 3. Test each tool against a benchmark harness (SWE-bench, Multi-SWE-bench).
• 4. Remove tools that fail to meet expectations (Headroom, LiteLLM).
• 5. Run pilot with remaining stack (Graphify, Serena, LeanCTX, Caveman) on 31 tasks.
• 6. Analyze token usage, correctness, and turn counts.
• 7. Conclude that token savings numbers are misleading without correctness checks.
• 8. Recommend reporting cost per solved task and turn count as sanity checks.
• 9. Publish findings and open-source repos for further work.

Доказательства

The experiment itself stopped making financial sense.

Источник

Token and cost savings numbers, without a real correctness check against the actual test suite, aren't just incomplete — they can point in exactly the wrong direction.

Источник