Mistral Large 4 Weights Are Out – What It Means for Building LLM Infrastructures

说明与用途

Mistral Large 4 是一个 7B 的开源 LLM，可在单个 24 GB GPU 上使用 Docker、vLLM 和 bitsandbytes 部署，提供 OpenAI 兼容 API 并支持 LangChain 集成。部署方案包含用于水平扩展的 Helm 图表。

问题

Deploy a 7‑B parameter LLM locally without high cloud costs or API limits.

结果

Mistral Large 4 can be deployed on a single 24 GB GPU with sub‑20 ms latency for 128‑token prompts, enabling cost‑effective, privacy‑first LLM services that integrate seamlessly with existing pipelines.

工作流

• Pull Docker image: docker pull mistralai/mistral-large-4:latest
• Run container with environment variables for model path, max tokens, and quantization: docker run -d --name mistral-llm -p 8000:8000 -e MODEL_PATH=/models/mistral-large-4 -e MAX_TOTAL_TOKENS=4096 -e QUANTIZATION=bitsandbytes mistralai/mistral-large-4:latest
• Test endpoint with curl to /v1/completions
• Integrate with LangChain by pointing OpenAI client to http://localhost:8000/v1 and using model_name "mistral-large-4"
• Scale horizontally with Helm chart: helm repo add mistral https://helm.mistral.ai; helm install mistral-large4 mistral/mistral-large4 --set replicaCount=3 --set resources.limits.memory=32Gi --set resources.limits.nvidia.com/gpu=1

证据

Mistral AI just open‑sourced the weights for Mistral Large 4, a 7‑billion‑parameter, dense transformer model that claims state‑of‑the‑art performance on a wide range of benchmarks.

来源

The release ships with a Hugging Face 🤗 transformers compatible checkpoint and a Docker image that includes an OpenAI‑compatible /v1/completions endpoint.

来源

The chart creates a Deployment with a sidecar that handles request routing and token caching.

来源