Deploying the 600GB Inkling-NVFP4 Model on Spot A3: A GKE and vLLM Deep Dive

说明与用途

使用官方 vLLM 镜像在 Spot A3 GKE 上部署 600GB Inkling‑NVFP4 模型，限制上下文为 4096 令牌，使用 96% VRAM，避免 ABI 冲突和 OOM。

问题

Running the 600GB Inkling-NVFP4 model on a Spot A3 GKE cluster crashes due to software conflicts and memory limits.

结果

The model runs successfully on Spot A3 with no crashes, using 96% VRAM headroom and a 4k token context window.

工作流

• 1. Replace Ray cluster image with official vLLM image (vllm/vllm-openai:v0.26.0) in Kubernetes YAML for head and workers.
• 2. Deploy the cluster on Spot A3 GKE instance.
• 3. Configure vLLM engine arguments: model="thinkingmachines/Inkling-NVFP4", tensor_parallel_size=8, gpu_memory_utilization=0.96, max_model_len=4096, trust_remote_code=True, enforce_eager=True.
• 4. Write a clean Python Ray Serve deployment script that initializes the engine and exposes /health and /completions endpoints.
• 5. Deploy the Python script to the cluster and verify the health endpoint returns success.
• 6. Send prompts to /completions and observe correct inference.
• 7. Monitor GPU memory usage to confirm no OOM occurs.

证据

By swapping out your Kubernetes YAML manifest to use this image for both the Head and Worker nodes, the servers boot up instantly with no API conflicts, no ABI mismatches, and zero download timeouts.

来源

Set max_model_len=4096 and gpu_memory_utilization=0.96 to fit the model and KV cache into VRAM.

来源