The Weekly Tally: 12 Rejected Outputs Told Me My Prompt Was the Bug, Not the Model

说明与用途

LLM拒绝的每周计数表明提示漂移是主要问题。实施提示差异仪式和固定重建减少了错误升级并简化了流程。

问题

Rejections of LLM outputs were largely caused by prompt drift rather than model or data issues.

结果

The process change reduced false escalations caused by fixture drift and clarified that most rejections were due to prompt drift, enabling focused fixes and preventing unnecessary model tuning.

工作流

• 1. Tag each rejected LLM output as prompt, fixture, model, or data.
• 2. Compile weekly tallies of tags per week.
• 3. Identify the tag with >50% of rejections as the primary bug source.
• 4. For prompt drift: perform a prompt diff review before each version bump.
• 5. For fixture drift: rebuild the fixture from a real ticket and rerun the suite.
• 6. For model drift: monitor trend over consecutive weeks before adjusting model settings.
• 7. Limit each tally review to one work order per Monday morning.

证据

Three weeks of review queues, 31 rejected outputs, one tag each:

来源

Prompt: 55%. Fixture: 23%. Model: 16%. Data: 6%.

来源

The SOP said one tag over 50% is where the bug lives.

来源

The work order: a prompt diff ritual before every version bump.

来源

The work order: I rebuilt the fixture from a real ticket, re-ran the suite, and the false escalations stopped.

来源