A Small Attention Fallback Fix Cut Peak Mask-Pass VRAM by 87%

Описание и применение

Переключение резервного пути внимания с MHA на SDPA в Xiaomi‑Lens‑0 снизило пиковой потребление памяти GPU на 87 % при работе с изображениями 3 МП, сохранив при этом точность выходных данных.

Задача

When an optional acceleration library is unavailable, the model’s fallback path consumes excessive GPU memory, causing failures on large images.

Результат

Switching the fallback from MHA to SDPA reduced peak GPU memory for the mask pass from 18.02 GB to 2.28 GB at 3 MP input, an 87 % decrease, while preserving output fidelity.

Рабочий процесс

• Identify that flash‑attn is unavailable and the decoder falls back to MHA.
• Measure peak GPU memory for the mask pass at various input sizes.
• Verify that the SDPA implementation exists and can be selected via XIAOMI_LENS_DECODER_ATTN=sdpa.
• Modify the routing logic to use SDPA as the fallback when flash‑attn fails.
• Re‑benchmark to confirm memory reduction.
• Validate outputs by comparing token IDs, boxes, and mask logits across six test cases.
• Ensure explicit MHA selection still produces identical results.
• Document the change and update warnings.

Доказательства

In Xiaomi‑Lens‑0, the pixel mask decoder fell back from flash‑attn to nn.MultiheadAttention. In the configuration I tested, that fallback made large images consume enough GPU memory to fail.

Источник

At a 3‑megapixel input size, the additional peak GPU memory used by the mask pass dropped from 18.02 GB to 2.28 GB—about 87%.

Источник

I compared six image‑and‑prompt cases, including object segmentation and a reasoning request, checking token IDs, boxes, and every mask logit.

Источник

The benchmark followed the inference stages used by infer.py. The table measures peak GPU memory added by run_forward, the mask pass, above the loaded‑model baseline.

Источник