Can You Trust an LLM to Fix a Broken Locator? I Measured It.

Описание и применение

LLM‑консенсус не может надёжно обнаруживать удалённые элементы: при единогласном выборе модели всегда происходит ложное исцеление. Поэтому в Automation Sandbox LLM используется только как опциональный, проверяемый через кворум, вспомогательный инструмент, а исцеления фиксируются только после успешного выполнения действия.

Задача

When a UI element is deleted during a refactor, a self‑healing engine must detect the absence and fail loudly instead of silently healing onto a neighboring element, which would hide a regression.

Результат

Unanimous LLM agreement on deleted elements is 100% wrong; disagreement is the only reliable indicator of absence. The framework therefore uses a deterministic scorer first, gates LLM use behind a quorum, and commits heals only after successful retries, ensuring safe self‑healing.

Рабочий процесс

• 1. Mutate a real UI tree by renaming, drifting, shifting, compound changes, and removing elements.
• 2. Run the deterministic scorer on all scenarios to establish baseline performance.
• 3. Invoke multiple LLM providers (3–7) with temperature 0 to vote on the target locator.
• 4. Record provider votes and agreement telemetry; consider a scenario usable only if ≥2 providers answer.
• 5. Analyze unanimous vs. non‑unanimous outcomes for surviving vs. removed elements.
• 6. Use results to design the final healing logic: deterministic first, LLM quorum‑gated, heal only after retry success, audit all decisions.

Доказательства

Providers reached unanimous agreement on 94.5% of scenarios where a successor existed, and on 43.6% where the element was gone.

Источник

Every one of the 52 unanimous verdicts on a surviving element was correct. Every one of the 34 unanimous verdicts on a deleted element was a false heal.

Источник