LLMeter: Measuring the Model You Actually Run

Описание и применение

LLMeter позволяет измерять производительность LLM на собственном оборудовании, учитывая загрузку весов, квантование и провайдеры, выдавая подробные отчёты и выявляя недостающие возможности модели.

Задача

Model leaderboards report performance on others’ hardware, not on a user’s own setup, making it hard to know how a model will run locally with specific quantization, GPU, and provider.

Результат

LLMeter measured a 2B model on a 6 GB GPU: first chat request took 64.28 s (due to weight loading), subsequent requests 328–730 ms, throughput 21.9–48.8 tokens/s. Consistency was perfect; structured‑output and tool‑calling failed, clearly reported. Reports are fully reproducible and versioned.

Рабочий процесс

• Install LLMeter via cargo or download binary
• Start the provider (e.g., Ollama) and load the model
• Run llmeter bench standard --model qwen3.5:2b --temperature 0.0 --max_tokens 16
• Review the generated reports (JSON, CSV, Markdown, HTML)
• Inspect first‑request latency, throughput, consistency, and capability flags
• Repeat runs for statistical confidence if needed
• Use llmeter bench perf sweep --dry-run to preview large matrix runs
• Integrate LLMeter commands into CI or automation scripts

Доказательства

LLMeter is a single Rust binary. You point it at an OpenAI-compatible /v1 endpoint and it measures what comes back.

Источник

The first number is the one worth looking at. Chat generation took 64.28 seconds to produce 16 tokens, an effective 0.25 tokens per second.

Источник

LLMeter produces a JSON record, a flattened CSV, a Markdown summary, and a self‑contained HTML report.

Источник