The biggest improvement in my skill evaluation came from a skill that was never invoked

Description and use

LLM skill evaluation can be misleading if skill invocation is not verified. The study found that apparent performance gains were due to random pass/fail variance, not actual skill use.

Problem

Evaluating LLM agent skills can produce misleading results if skill invocation is not verified, leading to false attribution of performance gains.

Outcome

The study revealed that apparent performance gains can arise from statistical noise rather than true skill execution, emphasizing the need to verify tool invocation when evaluating LLM agent skills.

Workflow

• 1. Run each skill twice: once with the built‑in eval, once with the custom runner. 2. Record pass/fail outcomes and continuous scores. 3. Inspect traces to confirm whether the Skill tool was invoked. 4. Compare results only if the skill was actually called. 5. Re‑evaluate tasks that reached the pass threshold in both arms to ensure they were challenging enough.

Evidence

Documentation and ADRs produced the largest apparent benefit in the whole run. Three of three tasks passed with the plugin, one of three without it.

Source

The Skill tool was not called in any of those three runs. I added a supplementary run to be sure. It was not called there either.

Source