Three settings that decide whether EmbeddingGemma 2 search works on your GPU

Описание и применение

Настройки dtype, ограничение потоков CPU и очистка кэша GPU критичны для корректной и быстрой работы EmbeddingGemma 2.

Задача

EmbeddingGemma 2 search performance and correctness depend on GPU data‑type, CPU thread limits, and GPU memory caching.

Результат

Correct dtype selection prevents silent embedding errors; CPU thread capping speeds image/video inference; clearing GPU cache restores VRAM for bulk indexing. Truncating to 256 dimensions offers storage savings with minimal quality loss.

Рабочий процесс

• 1. Load the model with dtype set to torch.bfloat16 if supported, otherwise torch.float32.
• 2. Determine the container’s CPU quota via cgroups and limit PyTorch thread count to that quota.
• 3. For large indexing batches, call torch.cuda.empty_cache() after processing to free unused GPU memory.
• 4. Optionally truncate embeddings to 256 dimensions and re‑normalize for storage efficiency.
• 5. Verify latency and VRAM usage per input type to ensure GPU sizing matches workload.

Доказательства

never float16: NaN or quietly wrong vectors, no exception

Источник

Limiting threads to the quota took a single image from 531 to 111 ms, eight images from 4,333 to 867 ms, and a 6-second video clip from 754 to 307 ms.

Источник

Calling torch.cuda.empty_cache() after very large requests fixed it: the same batch now leaves the process at 1.6 GB.

Источник