An AI Visibility Check Needs a Control Run, or You'll Report Noise as Movement

Описание и применение

Проверка видимости компаний в ответах LLM требует контрольного запуска: отсутствие компании стабильно, а список остальных имен меняется из‑за шума. Фокус аудита смещён на присутствие/отсутствие, а не на изменение списка.

Задача

Determining whether changes in vendor names returned by an LLM reflect real market shifts or are merely sampling noise.

Результат

The control run revealed that vendor list turnover is largely noise; absence of a company is a reliable metric. The firm shifted its audit claim to focus on presence/absence, improving credibility and reducing false positives.

Рабочий процесс

• 1. Prepare 13 buyer‑intent questions for each client.
• 2. Run each question three times with gemini‑flash‑latest, record raw outputs.
• 3. Extract vendor names from each run using a custom extractor.
• 4. Repeat steps 2–3 twenty minutes later (control batch).
• 5. Compare name lists from the two batches to compute symmetric difference.
• 6. Compare control batch to a batch run 17–20 days later to assess time‑based change.
• 7. If control difference ≈ time‑based difference, attribute changes to noise; otherwise report real shift.
• 8. Report only presence/absence of each client in the answer lists.
• 9. Publish findings and update client communications accordingly.

Доказательства

Both series are measured off the same extracted name lists, so the extraction defect described above inflates both arms equally and does not affect the comparison between them.

Источник

The mean symmetric difference — the count of names present in one reading and not the other — was 3.50 names across three weeks and 3.67 names across twenty minutes.

Источник

Absence was perfectly reproducible: the three companies we tested went unnamed in all 78 runs.

Источник