A third of the measurement went dark and the report stayed green

Description and use

One of three AI engines returned 404 due to a deprecated model, yet the citation report stayed at 0 %. After probing the provider, changing the default model, and updating summary logic, the system now correctly flags blind measurements, preventing silent outages.

Problem

A deprecated AI model caused one of three answer engines to return 404 for three weeks, but the monitoring report still showed 0% citations, masking the partial outage.

Outcome

The monitoring system now correctly distinguishes between zero measurements and blind measurements, preventing silent outages from going unnoticed and ensuring accurate visibility reports.

Workflow

• 1. Identify that openai-search engine returned 404 due to deprecated gpt-4o-search-preview.
• 2. Probe the provider directly instead of relying on configuration or catalog listings.
• 3. Change the default model in the tracker source to a non-deprecated model.
• 4. Update summarize() to name any engine that held a valid key but measured zero probes as BLIND.
• 5. Deploy changes, restart the process, and verify live run metrics.
• 6. Commit changes to main to ensure reproducibility.

Evidence

OpenAI deprecated gpt-4o-search-preview, so every request on the openai-search leg returned HTTP 404.

Source

Changed the default model in the tracker source rather than patching an env var on the box, so a rebuild cannot lose it.

Source