I measured my agent orchestrator against a plain agent. Here is what held up and what didn't.

Description and use

Stillwright adds auditability and prevents false claims, but on small tasks it does not improve efficiency and increases cost.

Problem

Demonstrate whether the Stillwright agent orchestrator adds value over a plain agent for small tasks, in terms of auditability, correctness, and cost.

Outcome

Stillwright added auditability and prevented false claims, but on small tasks it did not improve task completion or reduce cost; it increased wall time and token usage. After tweaking the skill, the five‑line inbox rule was satisfied and runtime dropped roughly to half of the initial Stillwright runs.

Workflow

• 1. Define three small tasks (code, writing, research).
• 2. Run each task twice: once with Stillwright skills, once without.
• 3. Grade results with scripts and a read‑only model judge.
• 4. Record wall time, token usage, and report length.
• 5. Identify shortcomings (five‑line rule, cost).
• 6. Modify skill text to enforce five‑line rule.
• 7. Re‑run Stillwright tasks and compare metrics.
• 8. Measure monetary cost with headless Claude sessions.
• 9. Publish results and tooling (sw run cost, sw cache).

Evidence

Three small, real tasks, each done twice by a fresh agent of the same model (Claude Sonnet): once told to follow the Stillwright skills, once with no skill at all.

Source

An auditable record. Every Stillwright run left a plan with the check for each step written before the work, and a log with the check's output. None of the plain runs left anything.

Source

It broke its own five‑line rule every time (8, 8 and 22 lines). The one‑line inbox cards were short; the report it returned was not.

Source

It costs more: about 1.7–2x the wall time and 10–20% more tokens, spent on the plan, the checks and the reviewer.

Source

All three now meet the rule with the proof still in the message. Wall time also dropped to about half of the plain runs' first‑round times; one re‑run is not enough to call that real.

Source