I tried to build a "token optimization stack" for coding agents. Here’s why I killed it.

说明与用途

作者尝试为Claude Code构建一个令牌优化堆栈，但试点结果显示令牌节省数字误导且存在正确性问题，因成本高昂和指标不可靠而放弃该项目。

问题

Token limits and high costs when using Claude Code for engineering tasks.

结果

The token-optimization stack produced misleading savings figures; a 97% token reduction corresponded to zero solved tasks. The pilot cost $5.60 per run and projected $1,200–$2,400 for full testing, with correctness failures on weak models.

工作流

• 1. Identify token-heavy areas: context reading, re-reading, verbose output, codebase navigation.
• 2. Assemble a stack of five tools: Graphify, Serena, Headroom, LiteLLM, Caveman.
• 3. Test each tool against a benchmark harness (SWE-bench, Multi-SWE-bench).
• 4. Remove tools that fail to meet expectations (Headroom, LiteLLM).
• 5. Run pilot with remaining stack (Graphify, Serena, LeanCTX, Caveman) on 31 tasks.
• 6. Analyze token usage, correctness, and turn counts.
• 7. Conclude that token savings numbers are misleading without correctness checks.
• 8. Recommend reporting cost per solved task and turn count as sanity checks.
• 9. Publish findings and open-source repos for further work.

证据

The experiment itself stopped making financial sense.

来源

Token and cost savings numbers, without a real correctness check against the actual test suite, aren't just incomplete — they can point in exactly the wrong direction.

来源