Part 2: We Fixed the Root Causes of Our AI Agent Incident. Routing Models by Step Cut Latency 86%

Description and use

The article explains how the team fixed AI agent issues by replacing legacy RAG, correcting SQL semantics, and optimizing routing, cutting latency from 42.6 s to 5.86 s and cost from $0.184 to $0.0318 per ticket.

Problem

An AI billing agent incorrectly routed a $48,000 credit to a human and incurred high latency and cost due to outdated retrieval and reasoning pipelines.

Outcome

Latency dropped from 42.6 s to 5.86 s (≈86% reduction) and cost from $0.184 to $0.0318 per ticket, while preserving correct billing logic.

Workflow

• 1. Replace fixed‑size chunking with section‑aware parent‑child indexing and hybrid search + reranker.
• 2. Add regression test to ensure notice clause retrieval after query rewrite.
• 3. Remove raw SQL generation; expose governed metrics via MCP tool.
• 4. Replace frontier models at boundaries 1,2,3,5,6 with fast instruct or deterministic gates.
• 5. Keep frontier model only at boundary 4 for multi‑clause policy adjudication.
• 6. Run replay tests to confirm latency and cost improvements.

Evidence

Failure 1: Fixed‑size chunking severed the condition from the formula

Source

We made three changes: Section‑aware parent‑child chunking… Hybrid search plus reranking… A regression test pinned to the incident.

Source

Once the replay tests were green, we looked at why TICKET‑8841 took 42.6 seconds and cost $0.184.

Source