Beyond Vibes: Architecting Closed-Loop AI Agents with Local SLMs, Deterministic Evals, and Human Learning Loops

Description and use

A local AI agent with deterministic verification and tri‑state guardrails eliminates math hallucinations and fabricated data in business drafts, providing safe output and continuous human‑in‑the‑loop learning.

Problem

Running a large language model locally for business workflows introduces math hallucinations, vague evaluation, lack of exit ramps, and stale human feedback.

Outcome

The closed‑loop architecture removed 100% of arithmetic errors, prevented fabricated IDs, and introduced graceful refusal and human escalation paths, while continuously learning from human edits.

Workflow

• 1. Receive draft request
• 2. Drafter Agent (Llama 3.2) generates draft
• 3. Hybrid Verifier runs deterministic checks (math, grounding, structure)
• 4. If checks fail, route to Revise with retry budget
• 5. If no billable data, route to Abstain and log gap
• 6. If verifier fails after retries, Escalate to Human Review Hub
• 7. Human approves or edits draft
• 8. Continuous Memory updates QWK evaluation harness with human feedback

Evidence

Language models are probabilistic token predictors, not algebraic solvers.

Source

By enforcing that every Python assertion must pass before the SLM tone check is even invoked, we eliminate 100% of mathematical errors at zero API cost.

Source

Injected‑Error test suite shows verifier catches math, missing, invalid, and ungrounded errors but misses semantic fabrications.

Source