Valid JSON, Wrong Answer: Semantic Checks for LLM Output in n8n

说明与用途

LLM 可能生成语法正确但语义错误的 JSON，导致 n8n 工作流执行错误。通过添加基于业务规则的第二层检查——跨字段一致性、可行性范围和确定性重新推导——可以检测这些错误并将其转交人工审核。

问题

LLM outputs can be syntactically valid JSON yet semantically incorrect, causing wrong actions in production workflows.

结果

Semantic checks catch errors that shape validation misses, reducing wrong actions and enabling a feedback loop for model improvement.

工作流

• 1. Receive customer message
• 2. LLM generates strict JSON
• 3. Schema node validates shape
• 4. If shape passes, run semantic rules:
• a. Cross‑field consistency checks (e.g., category vs. keywords in text)
• b. Plausibility ranges (e.g., amount within product price limits)
• c. Re‑derive disputed fields deterministically (regex extraction, keyword map)
• 5. If any rule fails, send item to review queue with original input, model answer, failed rule, and deterministic re‑derivation
• 6. Human reviews flagged items and resolves disagreements
• 7. Use resolved cases to build a test set for monthly model evaluation

证据

A schema node validated the shape: category was a string, priority was one of low, medium, high, amount was a number. Every field present, every type correct, zero validation errors for weeks.

来源

If category is refund but the message contains no money, no order number, and no word like "charged" or "refund," that is a contradiction, not a judgment call.

来源

Hard‑code the ranges your business actually has — product prices, valid regions, working hours — and reject anything outside them.

来源

When a check fails, the instinct is to retry the same prompt and hope for a better answer. Usually you get the same confident wrong answer back, at full token cost.

来源