The Weekly Tally: 12 Rejected Outputs Told Me My Prompt Was the Bug, Not the Model

Description and use

A weekly tally of LLM rejections revealed prompt drift as the main issue. Implementing prompt diff rituals and fixture rebuilds reduced false escalations and streamlined the workflow.

Problem

Rejections of LLM outputs were largely caused by prompt drift rather than model or data issues.

Outcome

The process change reduced false escalations caused by fixture drift and clarified that most rejections were due to prompt drift, enabling focused fixes and preventing unnecessary model tuning.

Workflow

• 1. Tag each rejected LLM output as prompt, fixture, model, or data.
• 2. Compile weekly tallies of tags per week.
• 3. Identify the tag with >50% of rejections as the primary bug source.
• 4. For prompt drift: perform a prompt diff review before each version bump.
• 5. For fixture drift: rebuild the fixture from a real ticket and rerun the suite.
• 6. For model drift: monitor trend over consecutive weeks before adjusting model settings.
• 7. Limit each tally review to one work order per Monday morning.

Evidence

Three weeks of review queues, 31 rejected outputs, one tag each:

Source

Prompt: 55%. Fixture: 23%. Model: 16%. Data: 6%.

Source

The SOP said one tag over 50% is where the bug lives.

Source

The work order: a prompt diff ritual before every version bump.

Source

The work order: I rebuilt the fixture from a real ticket, re-ran the suite, and the false escalations stopped.

Source