Valid JSON, Wrong Answer: Semantic Checks for LLM Output in n8n

Описание и применение

LLM может выдавать синтаксически корректный JSON, но семантически неверный, что приводит к ошибочным действиям в рабочих процессах n8n. Добавление второго уровня проверки на основе бизнес‑правил, кросс‑полевая консистентность и диапазоны допустимых значений позволяет выявлять такие ошибки и перенаправлять их на ручной контроль.

Задача

LLM outputs can be syntactically valid JSON yet semantically incorrect, causing wrong actions in production workflows.

Результат

Semantic checks catch errors that shape validation misses, reducing wrong actions and enabling a feedback loop for model improvement.

Рабочий процесс

• 1. Receive customer message
• 2. LLM generates strict JSON
• 3. Schema node validates shape
• 4. If shape passes, run semantic rules:
• a. Cross‑field consistency checks (e.g., category vs. keywords in text)
• b. Plausibility ranges (e.g., amount within product price limits)
• c. Re‑derive disputed fields deterministically (regex extraction, keyword map)
• 5. If any rule fails, send item to review queue with original input, model answer, failed rule, and deterministic re‑derivation
• 6. Human reviews flagged items and resolves disagreements
• 7. Use resolved cases to build a test set for monthly model evaluation

Доказательства

A schema node validated the shape: category was a string, priority was one of low, medium, high, amount was a number. Every field present, every type correct, zero validation errors for weeks.

Источник

If category is refund but the message contains no money, no order number, and no word like "charged" or "refund," that is a contradiction, not a judgment call.

Источник

Hard‑code the ranges your business actually has — product prices, valid regions, working hours — and reject anything outside them.

Источник

When a check fails, the instinct is to retry the same prompt and hope for a better answer. Usually you get the same confident wrong answer back, at full token cost.

Источник