I published a benchmark that was entirely my own bug

Description and use

A compression skill for LLMs produced fabricated tool calls and misleading benchmark metrics due to a harness bug that lost the neutral system prompt. After fixing the harness, the skill achieved higher fidelity and safety scores, eliminated hallucinations, and reduced agent‑to‑agent tokens by up to 59% while maintaining human‑facing token counts.

Problem

A compression skill for LLM responses produced fabricated tool calls and misleading metrics because the harness lost its framing system prompt, leading to false benchmark results.

Outcome

The corrected skill achieved higher correctness, fidelity, actionability, and safety scores, eliminated fabricated tool calls, and cut agent‑to‑agent tokens by up to 59% while keeping human‑facing token counts stable or slightly higher.

Workflow

• 1. Build summarization skill with folded details and a short non‑folded list. 2. Create a harness with 16 realistic scenarios, 3 runs each, two models. 3. Run blind grading with weighted rubric (correctness 30%, fidelity 25%, actionability 20%, safety 15%, concision 10%). 4. Identify instrumentation bug: only last --append-system-prompt is kept. 5. Fix harness to preserve neutral framing. 6. Re‑run evaluation and compare metrics. 7. Split skill into core and reference files to reduce per‑turn token cost. 8. Publish all runs, including failures, to the repo.

Evidence

Fourteen of 48 responses contained fabricated tool calls — the model writing out Read(...) and inventing the output.

Source

The bug was that Claude Code keeps only the last --append-system-prompt flag you pass it.

Source

On the corrected harness, Opus fabricates nothing. 0 of 48.

Source