I published a benchmark that was entirely my own bug

Описание и применение

Разработчик создал компрессионный модуль для LLM, но из‑за ошибки в инструменте оценки (потеря системного запроса) результаты были искажены, включая фальсифицированные вызовы инструментов. Исправив инструмент, модуль показал улучшенные показатели точности, достоверности и безопасности, сократив токены между агентами до 59 % и устранив ложные результаты.

Задача

A compression skill for LLM responses produced fabricated tool calls and misleading metrics because the harness lost its framing system prompt, leading to false benchmark results.

Результат

The corrected skill achieved higher correctness, fidelity, actionability, and safety scores, eliminated fabricated tool calls, and cut agent‑to‑agent tokens by up to 59% while keeping human‑facing token counts stable or slightly higher.

Рабочий процесс

• 1. Build summarization skill with folded details and a short non‑folded list. 2. Create a harness with 16 realistic scenarios, 3 runs each, two models. 3. Run blind grading with weighted rubric (correctness 30%, fidelity 25%, actionability 20%, safety 15%, concision 10%). 4. Identify instrumentation bug: only last --append-system-prompt is kept. 5. Fix harness to preserve neutral framing. 6. Re‑run evaluation and compare metrics. 7. Split skill into core and reference files to reduce per‑turn token cost. 8. Publish all runs, including failures, to the repo.

Доказательства

Fourteen of 48 responses contained fabricated tool calls — the model writing out Read(...) and inventing the output.

Источник

The bug was that Claude Code keeps only the last --append-system-prompt flag you pass it.

Источник

On the corrected harness, Opus fabricates nothing. 0 of 48.

Источник