Your AI agent can return HTTP 200 and still be failing

说明与用途

AI 代理可能返回 HTTP 200，但未完成任务，导致可靠性问题未被检测。ReliaMesh 展示了通过应用程序定义的结果监控来检测回归和恢复的方式。

问题

An AI agent may return HTTP 200 while failing to perform its intended task, leading to undetected application-level reliability issues.

结果

ReliaMesh demonstrates that monitoring application-level outcomes reveals regressions that HTTP status codes alone miss, enabling reproducible detection of reliability degradation and recovery.

工作流

• 1. Deploy AI agent to validate structured documents. 2. Record each validation event with outcome, model/prompt version, latency, token usage, and retry count. 3. Use ReliaMesh to compare adjacent windows of 50 events. 4. Detect regression if failure rate increases by ≥15 percentage points with non-overlapping Wilson confidence intervals. 5. Detect recovery when subsequent window shows lower failure rate. 6. Review detection logs to confirm reliability changes.

证据

An API returning HTTP 200 tells you that a request completed successfully. It doesn't tell you whether an AI agent actually did what it was supposed to do.

来源

ReliaMesh works with explicitly reported outcomes.

来源

For failure-rate regressions, it compares adjacent observation windows of 50 events.

来源

A regression signal requires a failure-rate increase of at least 15 percentage points, along with separation between the corresponding Wilson confidence intervals.

来源