I Tried to Measure That One Weird Failure Mode in [LLMs / Agents]

Описание и применение

Автор создал бенчмарк для измерения частоты и условий возникновения специфической ошибки в больших языковых моделях, что позволило оценить показатели моделей.

Задача

The author observed a recurring failure mode in large language models or agents during multi-step reasoning, code generation, or tool use, where the model confidently skips a step yet the final answer appears correct.

Результат

The benchmark enabled measurement of pass rates per model, revealing surprising differences and insights into the failure mode.

Рабочий процесс

• Designed test cases targeting the failure mode
• Defined objective scoring rules for failures
• Created a dataset of test cases
• Implemented tooling (e.g., Python scripts) to run tests
• Ran tests across selected models
• Scored responses using exact match or rubric
• Repeated runs to assess variance

Доказательства

I decided to build a benchmark for it.

Источник

Wrote [N] test cases that isolate the failure mode

Источник

Scored each response using [exact match / rubric / programmatic check]

Источник