I measured my agent's confidence. It was a constant.

Описание и применение

Fleet использует комбинацию модели Gemini и Gemma, а также эмпирические данные о подтверждениях и откатах, чтобы корректировать постоянные показатели уверенности и надёжно маршрутизировать задачи между агентами и людьми.

Задача

Agent confidence scores were constant and unreliable, leading to inappropriate routing of tasks to agents or humans.

Результат

Fleet achieved reliable task routing by combining model confidence with empirical human feedback, preventing constant confidence values from causing misrouting and ensuring agents only handle tasks they are proven capable of.

Рабочий процесс

• 1. Receive Slack message via /fleet or app mention. 2. Split message into discrete tasks. 3. Classify each task with Gemini 3.7 Flash. 4. Independently read each task with Gemma 4‑26B. 5. If models disagree, halve Gemini’s confidence. 6. Compute empirical rate: (approvals+2)/(approvals+send_backs+3). 7. Effective confidence = model_confidence * empirical_rate. 8. Route task based on tier thresholds (0.55, 0.75). 9. If routed to a person, record send‑back or approval. 10. Update empirical rate and repeat for future tasks.

Доказательства

The result was almost always 0.95.

Источник

effective_confidence = model_confidence * empirical_rate

Источник

If the models disagree, halve the first model’s confidence before deterministic routing.

Источник