Part 2: We Fixed the Root Causes of Our AI Agent Incident. Routing Models by Step Cut Latency 86%

说明与用途

文章说明团队如何通过替换旧的RAG、纠正SQL语义错误和优化路由，降低延迟从42.6秒到5.86秒，成本从0.184美元降至0.0318美元每个工单。

问题

An AI billing agent incorrectly routed a $48,000 credit to a human and incurred high latency and cost due to outdated retrieval and reasoning pipelines.

结果

Latency dropped from 42.6 s to 5.86 s (≈86% reduction) and cost from $0.184 to $0.0318 per ticket, while preserving correct billing logic.

工作流

• 1. Replace fixed‑size chunking with section‑aware parent‑child indexing and hybrid search + reranker.
• 2. Add regression test to ensure notice clause retrieval after query rewrite.
• 3. Remove raw SQL generation; expose governed metrics via MCP tool.
• 4. Replace frontier models at boundaries 1,2,3,5,6 with fast instruct or deterministic gates.
• 5. Keep frontier model only at boundary 4 for multi‑clause policy adjudication.
• 6. Run replay tests to confirm latency and cost improvements.

证据

Failure 1: Fixed‑size chunking severed the condition from the formula

来源

We made three changes: Section‑aware parent‑child chunking… Hybrid search plus reranking… A regression test pinned to the incident.

来源

Once the replay tests were green, we looked at why TICKET‑8841 took 42.6 seconds and cost $0.184.

来源