I measured my agent's confidence. It was a constant.

Description and use

Fleet combines Gemini and Gemma models with empirical approval data to adjust constant confidence scores, ensuring reliable routing of tasks between agents and humans.

Problem

Agent confidence scores were constant and unreliable, leading to inappropriate routing of tasks to agents or humans.

Outcome

Fleet achieved reliable task routing by combining model confidence with empirical human feedback, preventing constant confidence values from causing misrouting and ensuring agents only handle tasks they are proven capable of.

Workflow

• 1. Receive Slack message via /fleet or app mention. 2. Split message into discrete tasks. 3. Classify each task with Gemini 3.7 Flash. 4. Independently read each task with Gemma 4‑26B. 5. If models disagree, halve Gemini’s confidence. 6. Compute empirical rate: (approvals+2)/(approvals+send_backs+3). 7. Effective confidence = model_confidence * empirical_rate. 8. Route task based on tier thresholds (0.55, 0.75). 9. If routed to a person, record send‑back or approval. 10. Update empirical rate and repeat for future tasks.

Evidence

The result was almost always 0.95.

Source

effective_confidence = model_confidence * empirical_rate

Source

If the models disagree, halve the first model’s confidence before deterministic routing.

Source