Serving LFM2.5-24B-A2B on one 48 GB GPU: the flags, the traps and what to measure

Описание и применение

Настройка LFM2.5‑24B‑A2B на одном 48 ГБ GPU с использованием vLLM 0.23.0, правильными парсерами и без ограничения max_tokens, чтобы измерить задержку и пропускную способность.

Задача

Deploying the 24B‑parameter LFM2.5‑24B‑A2B model on a single 48 GB GPU while managing memory, parsing, and concurrency constraints.

Результат

The workflow enables LFM2.5‑24B‑A2B to run on a single 48 GB GPU with correct parsing and concurrency handling, preparing for empirical measurement of latency and throughput.

Рабочий процесс

• 1. Pin vLLM to 0.23.0 or newer to support LFM2.5.
• 2. Launch vLLM with command: `vllm serve LiquidAI/LFM2.5-24B-A2B-FP8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser lfm2`
• 3. Ensure max_tokens is unset unless a hard ceiling is needed.
• 4. Use default sampling config (temperature 0.1, top_k 50, repetition_penalty 1.05) or override via extra_body.
• 5. Run the provided bench.py script to measure first‑visible‑token latency, tokens per second, and concurrency effects.
• 6. Monitor VRAM usage: ~23 GB free after weights, plus vLLM overhead and KV cache (~2.7 GB per 131,072‑token conversation).

Доказательства

Liquid AI built LFM2.5-24B-A2B for agents on your own hardware. It has 24.0B parameters in total, the learned numbers that make up the model.

Источник

vLLM runs it natively from 0.23.0, but needs telling how to split that output: `vllm serve LiquidAI/LFM2.5-24B-A2B-FP8 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser lfm2`

Источник