The 10M Token Question

Описание и применение

Скрипт позволяет измерять токен‑потребление реальных задач, выявляя скрытые потери и ограничения инструментов, а таблица решений переводит метрики в практические действия.

Задача

Developers lack a practical way to measure token waste and performance of AI coding assistants in realistic, constrained environments.

Результат

The script enables developers to quantify token usage per real task, revealing hidden waste and tool limitations. By running multiple samples, variance is captured, and a decision table translates metrics into actionable steps, guiding tool selection and task refinement.

Рабочий процесс

• 1. Save the provided bash script as budget_test.sh.
• 2. Set ENDPOINT and API_KEY to point to your tool’s OpenAI‑compatible API or CLI wrapper.
• 3. Prepare a task file (e.g., task.md) describing a real coding issue.
• 4. Run the script three times: ./budget_test.sh task.md.
• 5. Inspect the generated JSONL log and aggregate metrics with jq.
• 6. Compare results against the decision table to diagnose issues.
• 7. Adjust task file, tool, or environment based on the diagnosis and repeat if needed.

Доказательства

"Give it a hard token budget, put it on a free server, and watch what it does while the meter is running."

Источник

"Run it three times on the same task, because one sample tells you nothing about variance, and then read the ledger like a doctor reads a chart."

Источник