Part 2: We Fixed the Root Causes of Our AI Agent Incident. Routing Models by Step Cut Latency 86%

Описание и применение

В статье описывается, как команда исправила проблемы в системе AI‑агента, заменив устаревший RAG‑поток, устранив ошибки SQL‑семантики и оптимизировав маршрутизацию, что сократило задержку с 42,6 с до 5,86 с и стоимость с $0,184 до $0,0318 за тикет.

Задача

An AI billing agent incorrectly routed a $48,000 credit to a human and incurred high latency and cost due to outdated retrieval and reasoning pipelines.

Результат

Latency dropped from 42.6 s to 5.86 s (≈86% reduction) and cost from $0.184 to $0.0318 per ticket, while preserving correct billing logic.

Рабочий процесс

• 1. Replace fixed‑size chunking with section‑aware parent‑child indexing and hybrid search + reranker.
• 2. Add regression test to ensure notice clause retrieval after query rewrite.
• 3. Remove raw SQL generation; expose governed metrics via MCP tool.
• 4. Replace frontier models at boundaries 1,2,3,5,6 with fast instruct or deterministic gates.
• 5. Keep frontier model only at boundary 4 for multi‑clause policy adjudication.
• 6. Run replay tests to confirm latency and cost improvements.

Доказательства

Failure 1: Fixed‑size chunking severed the condition from the formula

Источник

We made three changes: Section‑aware parent‑child chunking… Hybrid search plus reranking… A regression test pinned to the incident.

Источник

Once the replay tests were green, we looked at why TICKET‑8841 took 42.6 seconds and cost $0.184.

Источник