A third of the measurement went dark and the report stayed green

Описание и применение

Одна из трёх AI‑двигателей стала возвращать 404 из‑за устаревшей модели, но отчёт о цитировании показывал 0 %. После проверки провайдера, смены модели и обновления логики отчёта система теперь корректно сигнализирует о «слепых» измерениях, предотвращая незаметные сбои.

Задача

A deprecated AI model caused one of three answer engines to return 404 for three weeks, but the monitoring report still showed 0% citations, masking the partial outage.

Результат

The monitoring system now correctly distinguishes between zero measurements and blind measurements, preventing silent outages from going unnoticed and ensuring accurate visibility reports.

Рабочий процесс

• 1. Identify that openai-search engine returned 404 due to deprecated gpt-4o-search-preview.
• 2. Probe the provider directly instead of relying on configuration or catalog listings.
• 3. Change the default model in the tracker source to a non-deprecated model.
• 4. Update summarize() to name any engine that held a valid key but measured zero probes as BLIND.
• 5. Deploy changes, restart the process, and verify live run metrics.
• 6. Commit changes to main to ensure reproducibility.

Доказательства

OpenAI deprecated gpt-4o-search-preview, so every request on the openai-search leg returned HTTP 404.

Источник

Changed the default model in the tracker source rather than patching an env var on the box, so a rebuild cannot lose it.

Источник