Can You Trust an LLM to Fix a Broken Locator? I Measured It.

说明与用途

LLM共识无法可靠检测已删除的 UI 元素，统一同意总是导致错误修复。因此，Automation Sandbox 仅将 LLM 用作可选的、通过投票门槛的后备工具，只有在重试成功后才提交修复。

问题

When a UI element is deleted during a refactor, a self‑healing engine must detect the absence and fail loudly instead of silently healing onto a neighboring element, which would hide a regression.

结果

Unanimous LLM agreement on deleted elements is 100% wrong; disagreement is the only reliable indicator of absence. The framework therefore uses a deterministic scorer first, gates LLM use behind a quorum, and commits heals only after successful retries, ensuring safe self‑healing.

工作流

• 1. Mutate a real UI tree by renaming, drifting, shifting, compound changes, and removing elements.
• 2. Run the deterministic scorer on all scenarios to establish baseline performance.
• 3. Invoke multiple LLM providers (3–7) with temperature 0 to vote on the target locator.
• 4. Record provider votes and agreement telemetry; consider a scenario usable only if ≥2 providers answer.
• 5. Analyze unanimous vs. non‑unanimous outcomes for surviving vs. removed elements.
• 6. Use results to design the final healing logic: deterministic first, LLM quorum‑gated, heal only after retry success, audit all decisions.

证据

Providers reached unanimous agreement on 94.5% of scenarios where a successor existed, and on 43.6% where the element was gone.

来源

Every one of the 52 unanimous verdicts on a surviving element was correct. Every one of the 34 unanimous verdicts on a deleted element was a false heal.

来源