What does one token actually cost you?

说明与用途

作者开发了一个工具，用于在自托管 GPU 上准确计算令牌成本，揭示了高变异性和配置更改的影响。

问题

Self‑hosted LLM operators lack a reliable way to calculate the monetary cost per million tokens on their own hardware.

结果

The author discovered that a single server’s cost per million tokens fluctuated between $8.77 and $11.26 across four runs, and that a change to max_num_seqs from 1 to 8 on an A100 reduced cost from $0.746 to $0.234 per million tokens. The tool automates these measurements and only declares a winner when statistically significant.

工作流

• 1. Run a baseline measurement of tokens per second on the target model and GPU. 2. Record the hourly GPU cost. 3. Compute cost per million tokens using the formula: ($/hr ÷ 3600) ÷ tokens per second × 1,000,000. 4. Repeat the run multiple times to establish a confidence interval and noise floor. 5. Apply a configuration change (e.g., adjust max_num_seqs). 6. Re‑measure and compare results. 7. Use the tool to report CHEAPER, MORE EXPENSIVE, or NO WINNER based on statistical significance.

证据

$ per million tokens = ($/hr ÷ 3600) ÷ tokens per second × 1,000,000

来源

The same server doesn't give the same number twice.

来源

On an A100 with vLLM, moving max_num_seqs from 1 to 8 took output cost from $0.746 to $0.234 per million tokens, holding across six interleaved runs.

来源

Send the same prompts twice and a prefix cache makes run two look cheaper than it is.

来源

pipx install throttle check --url http://localhost:8000 --model <your-model> --gpu-hourly-rate <your $/hr>

来源