Tiny model. Big decisions. — How I built a 144M-parameter typed decision model that routes 82% of agent decisions off LLMs

Описание и применение

Модель 144 М параметров принимает решения по типизированным вопросам за 18 мс, уменьшая вызовы больших LLM на 82 % и повышая общую точность.

Задача

Agentic workflows spend most of their time on repetitive decision logic that is handled by large language models, incurring high cost and latency.

Результат

The lightweight model routes 82% of agent decisions locally, cutting LLM calls by 82% and slightly improving overall accuracy.

Рабочий процесс

• 1. Install the model and server via pip and download the weights locally.
• 2. Run the local server (phocinae.main).
• 3. Send a decision request to http://127.0.0.1:8155/v1/systemone with state and typed questions; receive deterministic verdicts with calibrated confidence.

Доказательства

Every agentic workflow I build hits the same wall: the interesting logic is 10 lines, and the other 90% is a language model deciding "allow or deny?", "which tool?", "pass or escape?" thousands of times a day, at chat‑model prices and chat‑model latency.

Источник

It cannot generate text. It takes a state plus a list of typed questions (yes/no, pick-one, 1-10 score) and returns, in one forward pass, a verdict per question with calibrated confidence.

Источник

GPU: 18.6 ms p50 per decision. CPU-only: ~1.5 s, no GPU at all.

Источник

With a τ=0.6 confidence gate, confident decisions stay local and the rest escape to a bigger model. Result on the zh route: LLM calls cut 82% (100% → 18%), while combined accuracy went 0.789 → 0.7948.

Источник

The server is local-only (127.0.0.1), pure PyTorch at runtime, and the protocol (/v1/systemone: noul / choice / score questions, calibrated probabilities) is fully specified in the repo.

Источник