I measured my agent's confidence. It was a constant.

说明与用途

Fleet 通过结合 Gemini 与 Gemma 模型以及人类批准的经验数据来调整恒定的置信度评分，确保任务在代理与人工之间可靠路由。

问题

Agent confidence scores were constant and unreliable, leading to inappropriate routing of tasks to agents or humans.

结果

Fleet achieved reliable task routing by combining model confidence with empirical human feedback, preventing constant confidence values from causing misrouting and ensuring agents only handle tasks they are proven capable of.

工作流

• 1. Receive Slack message via /fleet or app mention. 2. Split message into discrete tasks. 3. Classify each task with Gemini 3.7 Flash. 4. Independently read each task with Gemma 4‑26B. 5. If models disagree, halve Gemini’s confidence. 6. Compute empirical rate: (approvals+2)/(approvals+send_backs+3). 7. Effective confidence = model_confidence * empirical_rate. 8. Route task based on tier thresholds (0.55, 0.75). 9. If routed to a person, record send‑back or approval. 10. Update empirical rate and repeat for future tasks.

证据

The result was almost always 0.95.

来源

effective_confidence = model_confidence * empirical_rate

来源

If the models disagree, halve the first model’s confidence before deterministic routing.

来源