Valid JSON, Wrong Answer: Semantic Checks for LLM Output in n8n

Description and use

LLMs can output syntactically valid JSON that is semantically wrong, causing incorrect actions in n8n workflows. Adding a second layer of business‑rule‑based checks—cross‑field consistency, plausibility ranges, and deterministic re‑derivation—detects these errors and routes them to human review.

Problem

LLM outputs can be syntactically valid JSON yet semantically incorrect, causing wrong actions in production workflows.

Outcome

Semantic checks catch errors that shape validation misses, reducing wrong actions and enabling a feedback loop for model improvement.

Workflow

• 1. Receive customer message
• 2. LLM generates strict JSON
• 3. Schema node validates shape
• 4. If shape passes, run semantic rules:
• a. Cross‑field consistency checks (e.g., category vs. keywords in text)
• b. Plausibility ranges (e.g., amount within product price limits)
• c. Re‑derive disputed fields deterministically (regex extraction, keyword map)
• 5. If any rule fails, send item to review queue with original input, model answer, failed rule, and deterministic re‑derivation
• 6. Human reviews flagged items and resolves disagreements
• 7. Use resolved cases to build a test set for monthly model evaluation

Evidence

A schema node validated the shape: category was a string, priority was one of low, medium, high, amount was a number. Every field present, every type correct, zero validation errors for weeks.

Source

If category is refund but the message contains no money, no order number, and no word like "charged" or "refund," that is a contradiction, not a judgment call.

Source

Hard‑code the ranges your business actually has — product prices, valid regions, working hours — and reject anything outside them.

Source

When a check fails, the instinct is to retry the same prompt and hope for a better answer. Usually you get the same confident wrong answer back, at full token cost.

Source