Deploying the 600GB Inkling-NVFP4 Model on Spot A3: A GKE and vLLM Deep Dive

Description and use

Deploying the 600GB Inkling‑NVFP4 model on Spot A3 GKE with the official vLLM image avoids ABI conflicts and OOM by limiting context to 4096 tokens and using 96% VRAM.

Problem

Running the 600GB Inkling-NVFP4 model on a Spot A3 GKE cluster crashes due to software conflicts and memory limits.

Outcome

The model runs successfully on Spot A3 with no crashes, using 96% VRAM headroom and a 4k token context window.

Workflow

• 1. Replace Ray cluster image with official vLLM image (vllm/vllm-openai:v0.26.0) in Kubernetes YAML for head and workers.
• 2. Deploy the cluster on Spot A3 GKE instance.
• 3. Configure vLLM engine arguments: model="thinkingmachines/Inkling-NVFP4", tensor_parallel_size=8, gpu_memory_utilization=0.96, max_model_len=4096, trust_remote_code=True, enforce_eager=True.
• 4. Write a clean Python Ray Serve deployment script that initializes the engine and exposes /health and /completions endpoints.
• 5. Deploy the Python script to the cluster and verify the health endpoint returns success.
• 6. Send prompts to /completions and observe correct inference.
• 7. Monitor GPU memory usage to confirm no OOM occurs.

Evidence

By swapping out your Kubernetes YAML manifest to use this image for both the Head and Worker nodes, the servers boot up instantly with no API conflicts, no ABI mismatches, and zero download timeouts.

Source

Set max_model_len=4096 and gpu_memory_utilization=0.96 to fit the model and KV cache into VRAM.

Source