Can a Decision Model Replace Hand-Written Routing Rules? A 3,080-Message Experiment

Описание и применение

Эксперимент показал, что модель JEV превосходит ручные правила в точности и скорости, а LLM дешевле, но медленнее. Гибридный подход повышает точность, но увеличивает количество отказов.

Задача

Determine if a decision model can replace hand‑written routing rules for interpreting customer messages into concrete actions in a banking backend.

Результат

JEV outperformed the rule engine in accuracy and speed, while the LLM was cheaper but slower. The hybrid approach improved accuracy on answered messages but required more referrals. Overall, a decision model can replace hand‑written rules, but trade‑offs exist between accuracy, latency, cost, and human intervention.

Рабочий процесс

• 1. Build a fictional backend with 77 actions and 11 service modules.
• 2. Implement four dispatchers: rules, JEV, LLM, hybrid.
• 3. Use the Banking77 dataset (3,080 test messages) to evaluate exact‑action accuracy, abstentions, latency, and cost.
• 4. Measure metrics: accuracy, referral rate, median decision time, API cost, and router implementation lines.
• 5. Compare results and analyze trade‑offs between speed, cost, and accuracy.

Доказательства

JEV beat our rules and answered about nine times faster than the LLM. The LLM was cheaper.

Источник

Exact-action accuracy: Rules 65.5%, JEV 80.9%, LLM 81.5%, Hybrid 74.1%.

Источник

Median decision time: Rules 0.29 ms, JEV 111 ms, LLM 975 ms, Hybrid 112 ms.

Источник

API cost per 1,000: Rules $0, JEV $0.083, LLM $0.028, Hybrid $0.083.

Источник