Deploying the 600GB Inkling-NVFP4 Model on Spot A3: A GKE and vLLM Deep Dive

Описание и применение

Развертывание 600‑ГБ модели Inkling‑NVFP4 на Spot A3 GKE с использованием официального образа vLLM позволяет избежать конфликтов ABI и OOM, ограничив длину модели до 4096 токенов и использовав 96% VRAM.

Задача

Running the 600GB Inkling-NVFP4 model on a Spot A3 GKE cluster crashes due to software conflicts and memory limits.

Результат

The model runs successfully on Spot A3 with no crashes, using 96% VRAM headroom and a 4k token context window.

Рабочий процесс

• 1. Replace Ray cluster image with official vLLM image (vllm/vllm-openai:v0.26.0) in Kubernetes YAML for head and workers.
• 2. Deploy the cluster on Spot A3 GKE instance.
• 3. Configure vLLM engine arguments: model="thinkingmachines/Inkling-NVFP4", tensor_parallel_size=8, gpu_memory_utilization=0.96, max_model_len=4096, trust_remote_code=True, enforce_eager=True.
• 4. Write a clean Python Ray Serve deployment script that initializes the engine and exposes /health and /completions endpoints.
• 5. Deploy the Python script to the cluster and verify the health endpoint returns success.
• 6. Send prompts to /completions and observe correct inference.
• 7. Monitor GPU memory usage to confirm no OOM occurs.

Доказательства

By swapping out your Kubernetes YAML manifest to use this image for both the Head and Worker nodes, the servers boot up instantly with no API conflicts, no ABI mismatches, and zero download timeouts.

Источник

Set max_model_len=4096 and gpu_memory_utilization=0.96 to fit the model and KV cache into VRAM.

Источник