Three settings that decide whether EmbeddingGemma 2 search works on your GPU

Description and use

dtype choice, CPU thread limits, and GPU cache clearing are critical for correct and fast EmbeddingGemma 2 search.

Problem

EmbeddingGemma 2 search performance and correctness depend on GPU data‑type, CPU thread limits, and GPU memory caching.

Outcome

Correct dtype selection prevents silent embedding errors; CPU thread capping speeds image/video inference; clearing GPU cache restores VRAM for bulk indexing. Truncating to 256 dimensions offers storage savings with minimal quality loss.

Workflow

• 1. Load the model with dtype set to torch.bfloat16 if supported, otherwise torch.float32.
• 2. Determine the container’s CPU quota via cgroups and limit PyTorch thread count to that quota.
• 3. For large indexing batches, call torch.cuda.empty_cache() after processing to free unused GPU memory.
• 4. Optionally truncate embeddings to 256 dimensions and re‑normalize for storage efficiency.
• 5. Verify latency and VRAM usage per input type to ensure GPU sizing matches workload.

Evidence

never float16: NaN or quietly wrong vectors, no exception

Source

Limiting threads to the quota took a single image from 531 to 111 ms, eight images from 4,333 to 867 ms, and a 6-second video clip from 754 to 307 ms.

Source

Calling torch.cuda.empty_cache() after very large requests fixed it: the same batch now leaves the process at 1.6 GB.

Source