The biggest improvement in my skill evaluation came from a skill that was never invoked

Описание и применение

Оценка навыков LLM может вводить в заблуждение, если не проверять вызов навыка. В исследовании обнаружено, что видимый прирост производительности был результатом случайных успехов, а не реального использования навыка.

Задача

Evaluating LLM agent skills can produce misleading results if skill invocation is not verified, leading to false attribution of performance gains.

Результат

The study revealed that apparent performance gains can arise from statistical noise rather than true skill execution, emphasizing the need to verify tool invocation when evaluating LLM agent skills.

Рабочий процесс

• 1. Run each skill twice: once with the built‑in eval, once with the custom runner. 2. Record pass/fail outcomes and continuous scores. 3. Inspect traces to confirm whether the Skill tool was invoked. 4. Compare results only if the skill was actually called. 5. Re‑evaluate tasks that reached the pass threshold in both arms to ensure they were challenging enough.

Доказательства

Documentation and ADRs produced the largest apparent benefit in the whole run. Three of three tasks passed with the plugin, one of three without it.

Источник

The Skill tool was not called in any of those three runs. I added a supplementary run to be sure. It was not called there either.

Источник