What does one token actually cost you?

Description and use

The author built a tool to accurately calculate token cost on self‑hosted GPUs, revealing high variability and the impact of configuration changes.

Problem

Self‑hosted LLM operators lack a reliable way to calculate the monetary cost per million tokens on their own hardware.

Outcome

The author discovered that a single server’s cost per million tokens fluctuated between $8.77 and $11.26 across four runs, and that a change to max_num_seqs from 1 to 8 on an A100 reduced cost from $0.746 to $0.234 per million tokens. The tool automates these measurements and only declares a winner when statistically significant.

Workflow

• 1. Run a baseline measurement of tokens per second on the target model and GPU. 2. Record the hourly GPU cost. 3. Compute cost per million tokens using the formula: ($/hr ÷ 3600) ÷ tokens per second × 1,000,000. 4. Repeat the run multiple times to establish a confidence interval and noise floor. 5. Apply a configuration change (e.g., adjust max_num_seqs). 6. Re‑measure and compare results. 7. Use the tool to report CHEAPER, MORE EXPENSIVE, or NO WINNER based on statistical significance.

Evidence

$ per million tokens = ($/hr ÷ 3600) ÷ tokens per second × 1,000,000

Source

The same server doesn't give the same number twice.

Source

On an A100 with vLLM, moving max_num_seqs from 1 to 8 took output cost from $0.746 to $0.234 per million tokens, holding across six interleaved runs.

Source

Send the same prompts twice and a prefix cache makes run two look cheaper than it is.

Source

pipx install throttle check --url http://localhost:8000 --model <your-model> --gpu-hourly-rate <your $/hr>

Source