Can You Trust an LLM to Fix a Broken Locator? I Measured It.

Description and use

LLM consensus cannot reliably detect deleted UI elements; unanimous agreement always yields false heals. Thus, Automation Sandbox uses LLMs only as an optional, quorum‑gated fallback, committing heals only after successful retries.

Problem

When a UI element is deleted during a refactor, a self‑healing engine must detect the absence and fail loudly instead of silently healing onto a neighboring element, which would hide a regression.

Outcome

Unanimous LLM agreement on deleted elements is 100% wrong; disagreement is the only reliable indicator of absence. The framework therefore uses a deterministic scorer first, gates LLM use behind a quorum, and commits heals only after successful retries, ensuring safe self‑healing.

Workflow

• 1. Mutate a real UI tree by renaming, drifting, shifting, compound changes, and removing elements.
• 2. Run the deterministic scorer on all scenarios to establish baseline performance.
• 3. Invoke multiple LLM providers (3–7) with temperature 0 to vote on the target locator.
• 4. Record provider votes and agreement telemetry; consider a scenario usable only if ≥2 providers answer.
• 5. Analyze unanimous vs. non‑unanimous outcomes for surviving vs. removed elements.
• 6. Use results to design the final healing logic: deterministic first, LLM quorum‑gated, heal only after retry success, audit all decisions.

Evidence

Providers reached unanimous agreement on 94.5% of scenarios where a successor existed, and on 43.6% where the element was gone.

Source

Every one of the 52 unanimous verdicts on a surviving element was correct. Every one of the 34 unanimous verdicts on a deleted element was a false heal.

Source