Three settings that decide whether EmbeddingGemma 2 search works on your GPU

说明与用途

dtype 选择、CPU 线程限制和 GPU 缓存清理对 EmbeddingGemma 2 的正确性和速度至关重要。

问题

EmbeddingGemma 2 search performance and correctness depend on GPU data‑type, CPU thread limits, and GPU memory caching.

结果

Correct dtype selection prevents silent embedding errors; CPU thread capping speeds image/video inference; clearing GPU cache restores VRAM for bulk indexing. Truncating to 256 dimensions offers storage savings with minimal quality loss.

工作流

• 1. Load the model with dtype set to torch.bfloat16 if supported, otherwise torch.float32.
• 2. Determine the container’s CPU quota via cgroups and limit PyTorch thread count to that quota.
• 3. For large indexing batches, call torch.cuda.empty_cache() after processing to free unused GPU memory.
• 4. Optionally truncate embeddings to 256 dimensions and re‑normalize for storage efficiency.
• 5. Verify latency and VRAM usage per input type to ensure GPU sizing matches workload.

证据

never float16: NaN or quietly wrong vectors, no exception

来源

Limiting threads to the quota took a single image from 531 to 111 ms, eight images from 4,333 to 867 ms, and a 6-second video clip from 754 to 307 ms.

来源

Calling torch.cuda.empty_cache() after very large requests fixed it: the same batch now leaves the process at 1.6 GB.

来源