I published a benchmark that was entirely my own bug

说明与用途

开发者为 LLM 创建了压缩技能，但由于评估工具的错误导致系统提示丢失，产生了虚假工具调用和误导性指标。修复后，技能在准确性、真实性和安全性方面提升，消除了幻觉，并将代理间令牌减少至 59%，同时保持人类可读令牌数量。

问题

A compression skill for LLM responses produced fabricated tool calls and misleading metrics because the harness lost its framing system prompt, leading to false benchmark results.

结果

The corrected skill achieved higher correctness, fidelity, actionability, and safety scores, eliminated fabricated tool calls, and cut agent‑to‑agent tokens by up to 59% while keeping human‑facing token counts stable or slightly higher.

工作流

• 1. Build summarization skill with folded details and a short non‑folded list. 2. Create a harness with 16 realistic scenarios, 3 runs each, two models. 3. Run blind grading with weighted rubric (correctness 30%, fidelity 25%, actionability 20%, safety 15%, concision 10%). 4. Identify instrumentation bug: only last --append-system-prompt is kept. 5. Fix harness to preserve neutral framing. 6. Re‑run evaluation and compare metrics. 7. Split skill into core and reference files to reduce per‑turn token cost. 8. Publish all runs, including failures, to the repo.

证据

Fourteen of 48 responses contained fabricated tool calls — the model writing out Read(...) and inventing the output.

来源

The bug was that Claude Code keeps only the last --append-system-prompt flag you pass it.

来源

On the corrected harness, Opus fabricates nothing. 0 of 48.

来源