Beyond Vibes: Architecting Closed-Loop AI Agents with Local SLMs, Deterministic Evals, and Human Learning Loops

Описание и применение

Архитектура локального агента с три‑стейтовой защитой и детерминированными проверками устраняет математические ошибки и фальсификации в бизнес‑сообщениях, обеспечивая безопасный вывод и непрерывное обучение от человека.

Задача

Running a large language model locally for business workflows introduces math hallucinations, vague evaluation, lack of exit ramps, and stale human feedback.

Результат

The closed‑loop architecture removed 100% of arithmetic errors, prevented fabricated IDs, and introduced graceful refusal and human escalation paths, while continuously learning from human edits.

Рабочий процесс

• 1. Receive draft request
• 2. Drafter Agent (Llama 3.2) generates draft
• 3. Hybrid Verifier runs deterministic checks (math, grounding, structure)
• 4. If checks fail, route to Revise with retry budget
• 5. If no billable data, route to Abstain and log gap
• 6. If verifier fails after retries, Escalate to Human Review Hub
• 7. Human approves or edits draft
• 8. Continuous Memory updates QWK evaluation harness with human feedback

Доказательства

Language models are probabilistic token predictors, not algebraic solvers.

Источник

By enforcing that every Python assertion must pass before the SLM tone check is even invoked, we eliminate 100% of mathematical errors at zero API cost.

Источник

Injected‑Error test suite shows verifier catches math, missing, invalid, and ungrounded errors but misses semantic fabrications.

Источник