LLMeter: Measuring the Model You Actually Run

说明与用途

LLMeter 可在本地硬件上测量 LLM 性能，考虑权重加载、量化和提供商，生成详细报告并突出模型缺失功能。

问题

Model leaderboards report performance on others’ hardware, not on a user’s own setup, making it hard to know how a model will run locally with specific quantization, GPU, and provider.

结果

LLMeter measured a 2B model on a 6 GB GPU: first chat request took 64.28 s (due to weight loading), subsequent requests 328–730 ms, throughput 21.9–48.8 tokens/s. Consistency was perfect; structured‑output and tool‑calling failed, clearly reported. Reports are fully reproducible and versioned.

工作流

• Install LLMeter via cargo or download binary
• Start the provider (e.g., Ollama) and load the model
• Run llmeter bench standard --model qwen3.5:2b --temperature 0.0 --max_tokens 16
• Review the generated reports (JSON, CSV, Markdown, HTML)
• Inspect first‑request latency, throughput, consistency, and capability flags
• Repeat runs for statistical confidence if needed
• Use llmeter bench perf sweep --dry-run to preview large matrix runs
• Integrate LLMeter commands into CI or automation scripts

证据

LLMeter is a single Rust binary. You point it at an OpenAI-compatible /v1 endpoint and it measures what comes back.

来源

The first number is the one worth looking at. Chat generation took 64.28 seconds to produce 16 tokens, an effective 0.25 tokens per second.

来源

LLMeter produces a JSON record, a flattened CSV, a Markdown summary, and a self‑contained HTML report.

来源