Your AI agent can return HTTP 200 and still be failing

Description and use

An AI agent may return HTTP 200 while failing to perform its task, causing undetected reliability issues. ReliaMesh shows how application-defined outcome monitoring can detect regressions and recoveries that HTTP status alone misses.

Problem

An AI agent may return HTTP 200 while failing to perform its intended task, leading to undetected application-level reliability issues.

Outcome

ReliaMesh demonstrates that monitoring application-level outcomes reveals regressions that HTTP status codes alone miss, enabling reproducible detection of reliability degradation and recovery.

Workflow

• 1. Deploy AI agent to validate structured documents. 2. Record each validation event with outcome, model/prompt version, latency, token usage, and retry count. 3. Use ReliaMesh to compare adjacent windows of 50 events. 4. Detect regression if failure rate increases by ≥15 percentage points with non-overlapping Wilson confidence intervals. 5. Detect recovery when subsequent window shows lower failure rate. 6. Review detection logs to confirm reliability changes.

Evidence

An API returning HTTP 200 tells you that a request completed successfully. It doesn't tell you whether an AI agent actually did what it was supposed to do.

Source

ReliaMesh works with explicitly reported outcomes.

Source

For failure-rate regressions, it compares adjacent observation windows of 50 events.

Source

A regression signal requires a failure-rate increase of at least 15 percentage points, along with separation between the corresponding Wilson confidence intervals.

Source