Tiny model. Big decisions. — How I built a 144M-parameter typed decision model that routes 82% of agent decisions off LLMs

说明与用途

一款144M参数的模型以18ms做出类型化决策，减少82%的大型模型调用，并略微提升整体准确率。

问题

Agentic workflows spend most of their time on repetitive decision logic that is handled by large language models, incurring high cost and latency.

结果

The lightweight model routes 82% of agent decisions locally, cutting LLM calls by 82% and slightly improving overall accuracy.

工作流

• 1. Install the model and server via pip and download the weights locally.
• 2. Run the local server (phocinae.main).
• 3. Send a decision request to http://127.0.0.1:8155/v1/systemone with state and typed questions; receive deterministic verdicts with calibrated confidence.

证据

Every agentic workflow I build hits the same wall: the interesting logic is 10 lines, and the other 90% is a language model deciding "allow or deny?", "which tool?", "pass or escape?" thousands of times a day, at chat‑model prices and chat‑model latency.

来源

It cannot generate text. It takes a state plus a list of typed questions (yes/no, pick-one, 1-10 score) and returns, in one forward pass, a verdict per question with calibrated confidence.

来源

GPU: 18.6 ms p50 per decision. CPU-only: ~1.5 s, no GPU at all.

来源

With a τ=0.6 confidence gate, confident decisions stay local and the rest escape to a bigger model. Result on the zh route: LLM calls cut 82% (100% → 18%), while combined accuracy went 0.789 → 0.7948.

来源

The server is local-only (127.0.0.1), pure PyTorch at runtime, and the protocol (/v1/systemone: noul / choice / score questions, calibrated probabilities) is fully specified in the repo.

来源