A Small Attention Fallback Fix Cut Peak Mask-Pass VRAM by 87%

说明与用途

将 Xiaomi‑Lens‑0 的备用注意力路径从 MHA 切换到 SDPA，降低了 3 MP 图像的峰值 GPU 内存使用量 87%，同时保持了输出质量。

问题

When an optional acceleration library is unavailable, the model’s fallback path consumes excessive GPU memory, causing failures on large images.

结果

Switching the fallback from MHA to SDPA reduced peak GPU memory for the mask pass from 18.02 GB to 2.28 GB at 3 MP input, an 87 % decrease, while preserving output fidelity.

工作流

• Identify that flash‑attn is unavailable and the decoder falls back to MHA.
• Measure peak GPU memory for the mask pass at various input sizes.
• Verify that the SDPA implementation exists and can be selected via XIAOMI_LENS_DECODER_ATTN=sdpa.
• Modify the routing logic to use SDPA as the fallback when flash‑attn fails.
• Re‑benchmark to confirm memory reduction.
• Validate outputs by comparing token IDs, boxes, and mask logits across six test cases.
• Ensure explicit MHA selection still produces identical results.
• Document the change and update warnings.

证据

In Xiaomi‑Lens‑0, the pixel mask decoder fell back from flash‑attn to nn.MultiheadAttention. In the configuration I tested, that fallback made large images consume enough GPU memory to fail.

来源

At a 3‑megapixel input size, the additional peak GPU memory used by the mask pass dropped from 18.02 GB to 2.28 GB—about 87%.

来源

I compared six image‑and‑prompt cases, including object segmentation and a reasoning request, checking token IDs, boxes, and every mask logit.

来源

The benchmark followed the inference stages used by infer.py. The table measures peak GPU memory added by run_forward, the mask pass, above the loaded‑model baseline.

来源