An AI Visibility Check Needs a Control Run, or You'll Report Noise as Movement

Description and use

AI visibility checks need a control run: absence of a company is stable, while the rest of the list fluctuates due to noise. The audit focus shifts to presence/absence rather than list turnover.

Problem

Determining whether changes in vendor names returned by an LLM reflect real market shifts or are merely sampling noise.

Outcome

The control run revealed that vendor list turnover is largely noise; absence of a company is a reliable metric. The firm shifted its audit claim to focus on presence/absence, improving credibility and reducing false positives.

Workflow

• 1. Prepare 13 buyer‑intent questions for each client.
• 2. Run each question three times with gemini‑flash‑latest, record raw outputs.
• 3. Extract vendor names from each run using a custom extractor.
• 4. Repeat steps 2–3 twenty minutes later (control batch).
• 5. Compare name lists from the two batches to compute symmetric difference.
• 6. Compare control batch to a batch run 17–20 days later to assess time‑based change.
• 7. If control difference ≈ time‑based difference, attribute changes to noise; otherwise report real shift.
• 8. Report only presence/absence of each client in the answer lists.
• 9. Publish findings and update client communications accordingly.

Evidence

Both series are measured off the same extracted name lists, so the extraction defect described above inflates both arms equally and does not affect the comparison between them.

Source

The mean symmetric difference — the count of names present in one reading and not the other — was 3.50 names across three weeks and 3.67 names across twenty minutes.

Source

Absence was perfectly reproducible: the three companies we tested went unnamed in all 78 runs.

Source