Tiny model. Big decisions. — How I built a 144M-parameter typed decision model that routes 82% of agent decisions off LLMs

Description and use

A 144 M‑parameter model makes typed decisions in 18 ms, cutting large‑model calls by 82 % and slightly improving overall accuracy.

Problem

Agentic workflows spend most of their time on repetitive decision logic that is handled by large language models, incurring high cost and latency.

Outcome

The lightweight model routes 82% of agent decisions locally, cutting LLM calls by 82% and slightly improving overall accuracy.

Workflow

• 1. Install the model and server via pip and download the weights locally.
• 2. Run the local server (phocinae.main).
• 3. Send a decision request to http://127.0.0.1:8155/v1/systemone with state and typed questions; receive deterministic verdicts with calibrated confidence.

Evidence

Every agentic workflow I build hits the same wall: the interesting logic is 10 lines, and the other 90% is a language model deciding "allow or deny?", "which tool?", "pass or escape?" thousands of times a day, at chat‑model prices and chat‑model latency.

Source

It cannot generate text. It takes a state plus a list of typed questions (yes/no, pick-one, 1-10 score) and returns, in one forward pass, a verdict per question with calibrated confidence.

Source

GPU: 18.6 ms p50 per decision. CPU-only: ~1.5 s, no GPU at all.

Source

With a τ=0.6 confidence gate, confident decisions stay local and the rest escape to a bigger model. Result on the zh route: LLM calls cut 82% (100% → 18%), while combined accuracy went 0.789 → 0.7948.

Source

The server is local-only (127.0.0.1), pure PyTorch at runtime, and the protocol (/v1/systemone: noul / choice / score questions, calibrated probabilities) is fully specified in the repo.

Source