Your AI agent can return HTTP 200 and still be failing

Описание и применение

AI‑агент может возвращать HTTP 200, но при этом не выполнять задачу, что приводит к неотслеживаемым сбоям. ReliaMesh демонстрирует, как мониторинг результатов, определяемых приложением, выявляет регрессии и восстановление, недоступные при проверке только статуса ответа.

Задача

An AI agent may return HTTP 200 while failing to perform its intended task, leading to undetected application-level reliability issues.

Результат

ReliaMesh demonstrates that monitoring application-level outcomes reveals regressions that HTTP status codes alone miss, enabling reproducible detection of reliability degradation and recovery.

Рабочий процесс

• 1. Deploy AI agent to validate structured documents. 2. Record each validation event with outcome, model/prompt version, latency, token usage, and retry count. 3. Use ReliaMesh to compare adjacent windows of 50 events. 4. Detect regression if failure rate increases by ≥15 percentage points with non-overlapping Wilson confidence intervals. 5. Detect recovery when subsequent window shows lower failure rate. 6. Review detection logs to confirm reliability changes.

Доказательства

An API returning HTTP 200 tells you that a request completed successfully. It doesn't tell you whether an AI agent actually did what it was supposed to do.

Источник

ReliaMesh works with explicitly reported outcomes.

Источник

For failure-rate regressions, it compares adjacent observation windows of 50 events.

Источник

A regression signal requires a failure-rate increase of at least 15 percentage points, along with separation between the corresponding Wilson confidence intervals.

Источник