Stop Chasing Perfect Prompts: Build Systems Instead

Описание и применение

Проблема: отсутствие чётких правил и обработки ошибок в пайплайне LLM приводило к непредсказуемым оценкам и сбоям при реальных данных. Решение: создать отдельный рубрик оценивания, добавить жёсткие ограничения, очередь ошибок, централизованную конфигурацию и чёткие контракты между этапами. Итог: снижение ошибок, корректировка оценок и ускорение отладки.

Задача

LLM pipelines lacked explicit scoring rules and error handling, causing inconsistent confidence scores and failures when real data was used.

Результат

Explicit rule definition and system design reduced dead letter rate from 41% to 11%, corrected confidence scores immediately, and cut debugging time from hours to minutes.

Рабочий процесс

• 1. Define a scoring rubric as a separate document outlining evidence weights and thresholds.
• 2. Translate the rubric into a system instruction with hard constraints and explicit consequences.
• 3. Add a dead letter queue to capture failures before feature completion.
• 4. Centralize configuration (thresholds, endpoints) in a single loader.
• 5. Chain outputs between stages with defined contracts (e.g., web search results to scoring node).
• 6. Run real‑data validation before production deployment.
• 7. Instrument the pipeline to measure token usage and costs.
• 8. Iterate on optimization based on measured baselines.

Доказательства

We added four lines: what the hint represents, how to cross‑reference it against web search results, how a confirmed match shifts the threshold, and what to do when no hint arrives. Scores corrected immediately.

Источник

With it, you see a 41% dead letter rate drop to 11% after a single parser fix, because the queue made the failure visible.

Источник