What does one token actually cost you?

Описание и применение

Автор разработал инструмент для точного расчёта стоимости токенов на собственном GPU, выявив значительную вариабельность и влияние настроек.

Задача

Self‑hosted LLM operators lack a reliable way to calculate the monetary cost per million tokens on their own hardware.

Результат

The author discovered that a single server’s cost per million tokens fluctuated between $8.77 and $11.26 across four runs, and that a change to max_num_seqs from 1 to 8 on an A100 reduced cost from $0.746 to $0.234 per million tokens. The tool automates these measurements and only declares a winner when statistically significant.

Рабочий процесс

• 1. Run a baseline measurement of tokens per second on the target model and GPU. 2. Record the hourly GPU cost. 3. Compute cost per million tokens using the formula: ($/hr ÷ 3600) ÷ tokens per second × 1,000,000. 4. Repeat the run multiple times to establish a confidence interval and noise floor. 5. Apply a configuration change (e.g., adjust max_num_seqs). 6. Re‑measure and compare results. 7. Use the tool to report CHEAPER, MORE EXPENSIVE, or NO WINNER based on statistical significance.

Доказательства

$ per million tokens = ($/hr ÷ 3600) ÷ tokens per second × 1,000,000

Источник

The same server doesn't give the same number twice.

Источник

On an A100 with vLLM, moving max_num_seqs from 1 to 8 took output cost from $0.746 to $0.234 per million tokens, holding across six interleaved runs.

Источник

Send the same prompts twice and a prefix cache makes run two look cheaper than it is.

Источник

pipx install throttle check --url http://localhost:8000 --model <your-model> --gpu-hourly-rate <your $/hr>

Источник