A Small Attention Fallback Fix Cut Peak Mask-Pass VRAM by 87%

Description and use

Switching the fallback attention path from MHA to SDPA in Xiaomi‑Lens‑0 reduced peak GPU memory usage by 87 % for 3 MP images while maintaining output fidelity.

Problem

When an optional acceleration library is unavailable, the model’s fallback path consumes excessive GPU memory, causing failures on large images.

Outcome

Switching the fallback from MHA to SDPA reduced peak GPU memory for the mask pass from 18.02 GB to 2.28 GB at 3 MP input, an 87 % decrease, while preserving output fidelity.

Workflow

• Identify that flash‑attn is unavailable and the decoder falls back to MHA.
• Measure peak GPU memory for the mask pass at various input sizes.
• Verify that the SDPA implementation exists and can be selected via XIAOMI_LENS_DECODER_ATTN=sdpa.
• Modify the routing logic to use SDPA as the fallback when flash‑attn fails.
• Re‑benchmark to confirm memory reduction.
• Validate outputs by comparing token IDs, boxes, and mask logits across six test cases.
• Ensure explicit MHA selection still produces identical results.
• Document the change and update warnings.

Evidence

In Xiaomi‑Lens‑0, the pixel mask decoder fell back from flash‑attn to nn.MultiheadAttention. In the configuration I tested, that fallback made large images consume enough GPU memory to fail.

Source

At a 3‑megapixel input size, the additional peak GPU memory used by the mask pass dropped from 18.02 GB to 2.28 GB—about 87%.

Source

I compared six image‑and‑prompt cases, including object segmentation and a reasoning request, checking token IDs, boxes, and every mask logit.

Source

The benchmark followed the inference stages used by infer.py. The table measures peak GPU memory added by run_forward, the mask pass, above the loaded‑model baseline.

Source