LLMeter: Measuring the Model You Actually Run

Description and use

LLMeter measures LLM performance on your own hardware, accounting for weight loading, quantization, and providers, producing detailed reports and highlighting missing model capabilities.

Problem

Model leaderboards report performance on others’ hardware, not on a user’s own setup, making it hard to know how a model will run locally with specific quantization, GPU, and provider.

Outcome

LLMeter measured a 2B model on a 6 GB GPU: first chat request took 64.28 s (due to weight loading), subsequent requests 328–730 ms, throughput 21.9–48.8 tokens/s. Consistency was perfect; structured‑output and tool‑calling failed, clearly reported. Reports are fully reproducible and versioned.

Workflow

• Install LLMeter via cargo or download binary
• Start the provider (e.g., Ollama) and load the model
• Run llmeter bench standard --model qwen3.5:2b --temperature 0.0 --max_tokens 16
• Review the generated reports (JSON, CSV, Markdown, HTML)
• Inspect first‑request latency, throughput, consistency, and capability flags
• Repeat runs for statistical confidence if needed
• Use llmeter bench perf sweep --dry-run to preview large matrix runs
• Integrate LLMeter commands into CI or automation scripts

Evidence

LLMeter is a single Rust binary. You point it at an OpenAI-compatible /v1 endpoint and it measures what comes back.

Source

The first number is the one worth looking at. Chat generation took 64.28 seconds to produce 16 tokens, an effective 0.25 tokens per second.

Source

LLMeter produces a JSON record, a flattened CSV, a Markdown summary, and a self‑contained HTML report.

Source