The biggest improvement in my skill evaluation came from a skill that was never invoked
说明与用途
LLM 技能评估如果不验证技能调用,可能导致误导性结果。研究发现,表面上的性能提升是随机通过率导致的,而非实际技能使用。
问题
Evaluating LLM agent skills can produce misleading results if skill invocation is not verified, leading to false attribution of performance gains.
结果
The study revealed that apparent performance gains can arise from statistical noise rather than true skill execution, emphasizing the need to verify tool invocation when evaluating LLM agent skills.
工作流
- 1. Run each skill twice: once with the built‑in eval, once with the custom runner. 2. Record pass/fail outcomes and continuous scores. 3. Inspect traces to confirm whether the Skill tool was invoked. 4. Compare results only if the skill was actually called. 5. Re‑evaluate tasks that reached the pass threshold in both arms to ensure they were challenging enough.
证据
Documentation and ADRs produced the largest apparent benefit in the whole run. Three of three tasks passed with the plugin, one of three without it.
The Skill tool was not called in any of those three runs. I added a supplementary run to be sure. It was not called there either.
- 记录日期: 2026-09-22T07:15:23.006Z