Stop Chasing Perfect Prompts: Build Systems Instead

说明与用途

问题：LLM管道缺乏明确的评分规则和错误处理，导致信心分数不一致并在真实数据上出现失败。解决方案：制定单独的评分准则，添加硬约束、错误队列、集中配置和清晰的阶段间契约。结果：错误率下降，评分准确，调试更快。

问题

LLM pipelines lacked explicit scoring rules and error handling, causing inconsistent confidence scores and failures when real data was used.

结果

Explicit rule definition and system design reduced dead letter rate from 41% to 11%, corrected confidence scores immediately, and cut debugging time from hours to minutes.

工作流

• 1. Define a scoring rubric as a separate document outlining evidence weights and thresholds.
• 2. Translate the rubric into a system instruction with hard constraints and explicit consequences.
• 3. Add a dead letter queue to capture failures before feature completion.
• 4. Centralize configuration (thresholds, endpoints) in a single loader.
• 5. Chain outputs between stages with defined contracts (e.g., web search results to scoring node).
• 6. Run real‑data validation before production deployment.
• 7. Instrument the pipeline to measure token usage and costs.
• 8. Iterate on optimization based on measured baselines.

证据

We added four lines: what the hint represents, how to cross‑reference it against web search results, how a confirmed match shifts the threshold, and what to do when no hint arrives. Scores corrected immediately.

来源

With it, you see a 41% dead letter rate drop to 11% after a single parser fix, because the queue made the failure visible.

来源