Mistral Large 4 open weights: can you run it on your own servers?

Description and use

Organizations can host Mistral Large 4 on their own servers, but it requires 4–8 modern GPUs, compliance with the license, and offline component configuration.

Problem

Can an organization host Mistral Large 4 on its own servers without sending data to an API, and what are the hardware, licensing, and operational requirements?

Outcome

Self‑hosting Mistral Large 4 is feasible but requires substantial GPU resources, careful offline configuration, and compliance with a commercial license for third‑party service. Organizations can achieve sovereign AI while managing cost and operational complexity.

Workflow

• 1. Verify model availability and license terms on Hugging Face.
• 2. Assess hardware needs: 4–8 latest‑generation GPUs (e.g., Nvidia B200/B300 Blackwell) for NVFP4 or 8 GPUs for bf16 with tensor parallelism.
• 3. Install vLLM and required dependencies, ensuring local tokenizer files are accessible (fix for mistral‑common bug).
• 4. Configure KV cache, autoscaling, and monitoring per vLLM best practices.
• 5. Evaluate the model against internal use cases and compare with smaller models and the API.
• 6. Add custom guardrails and compliance checks, since API protections do not apply.
• 7. Obtain a commercial license from Mistral if the model will be offered as a service to third parties.

Evidence

The weights are about 1.06 TB in bf16 and 610 GB in NVFP4. The published vLLM recipe uses eight GPUs with tensor parallelism. The NVFP4 version runs, according to Mistral, on four Nvidia B200 or B300 GPUs (Blackwell).

Source

The Mistral Ecosystem Open-Weight License is a modified MIT licence. Internal use, commercial use and embedding the model in a product are free, with no revenue threshold. Selling access to the model as a service requires a separate commercial licence with Mistral.

Source

I fixed a bug in mistral-common that prevented loading a tokenizer already on disk.

Source