Validating LLM Output in n8n

说明与用途

在 n8n 工作流中，LLM 输出的 JSON 语法正确但语义错误，导致消息错误路由。实施了两步验证：严格的模式验证和语义检查。错误进入审核队列而非重试。原始输出被记录，真实错误成为回归测试用例。

问题

An n8n workflow that classifies incoming messages via an LLM returned valid JSON that was semantically incorrect, causing mis‑routing and priority errors.

结果

The workflow now reliably rejects semantically incorrect JSON, preventing mis‑routing and priority errors while maintaining fast deterministic validation and a review process for corrections.

工作流

• 1. Send message to LLM and request JSON output
• 2. Parse JSON with strict schema validation
• 3. Perform meaning checks on each field (category, priority, dates, IDs)
• 4. If any check fails, route to review queue with raw output, errors, and original input
• 5. For format‑only failures, run a deterministic cleanup pass before review
• 6. Log raw LLM output on every run
• 7. Add each real failure to regression fixture set
• 8. Run fixtures on every validation change to catch drift

证据

The JSON node never complained. The workflow never errored. And once or twice a week, a message landed in the wrong queue, or with a priority that made no sense, or with a summary describing a different message entirely.

来源

Parsing is not validation. If your workflow treats "it parsed" as "it is correct", you have no validation at all — you have a parser with good manners.

来源

When validation fails, the tempting move is to retry the model call. ... a model that invents a category will often invent it again, slightly differently, and each retry is another chance for a plausible‑looking wrong answer to slip through.

来源

The most useful debugging artifact in these workflows is the raw model output, before any parsing, cleanup, or transformation.

来源

Every validation failure that reaches review becomes a fixture: the raw output, the expected verdict, and the reason.

来源