The Weekly Tally: 12 Rejected Outputs Told Me My Prompt Was the Bug, Not the Model

Описание и применение

Система еженедельного подсчёта отказов LLM выявила, что большинство ошибок вызваны дрейфом подсказок, а не моделью. Внедрение дифф-ритуала подсказок и пересборки фикстур снизило ложные эскалации и улучшило процесс.

Задача

Rejections of LLM outputs were largely caused by prompt drift rather than model or data issues.

Результат

The process change reduced false escalations caused by fixture drift and clarified that most rejections were due to prompt drift, enabling focused fixes and preventing unnecessary model tuning.

Рабочий процесс

• 1. Tag each rejected LLM output as prompt, fixture, model, or data.
• 2. Compile weekly tallies of tags per week.
• 3. Identify the tag with >50% of rejections as the primary bug source.
• 4. For prompt drift: perform a prompt diff review before each version bump.
• 5. For fixture drift: rebuild the fixture from a real ticket and rerun the suite.
• 6. For model drift: monitor trend over consecutive weeks before adjusting model settings.
• 7. Limit each tally review to one work order per Monday morning.

Доказательства

Three weeks of review queues, 31 rejected outputs, one tag each:

Источник

Prompt: 55%. Fixture: 23%. Model: 16%. Data: 6%.

Источник

The SOP said one tag over 50% is where the bug lives.

Источник

The work order: a prompt diff ritual before every version bump.

Источник

The work order: I rebuilt the fixture from a real ticket, re-ran the suite, and the false escalations stopped.

Источник