Validating LLM Output in n8n

Description and use

In an n8n workflow, an LLM produced syntactically valid but semantically incorrect JSON, causing mis‑routing. A two‑step validation—strict schema then meaning checks—was added. Failures go to a review queue instead of retrying. Raw outputs are logged and real failures become regression fixtures.

Problem

An n8n workflow that classifies incoming messages via an LLM returned valid JSON that was semantically incorrect, causing mis‑routing and priority errors.

Outcome

The workflow now reliably rejects semantically incorrect JSON, preventing mis‑routing and priority errors while maintaining fast deterministic validation and a review process for corrections.

Workflow

• 1. Send message to LLM and request JSON output
• 2. Parse JSON with strict schema validation
• 3. Perform meaning checks on each field (category, priority, dates, IDs)
• 4. If any check fails, route to review queue with raw output, errors, and original input
• 5. For format‑only failures, run a deterministic cleanup pass before review
• 6. Log raw LLM output on every run
• 7. Add each real failure to regression fixture set
• 8. Run fixtures on every validation change to catch drift

Evidence

The JSON node never complained. The workflow never errored. And once or twice a week, a message landed in the wrong queue, or with a priority that made no sense, or with a summary describing a different message entirely.

Source

Parsing is not validation. If your workflow treats "it parsed" as "it is correct", you have no validation at all — you have a parser with good manners.

Source

When validation fails, the tempting move is to retry the model call. ... a model that invents a category will often invent it again, slightly differently, and each retry is another chance for a plausible‑looking wrong answer to slip through.

Source

The most useful debugging artifact in these workflows is the raw model output, before any parsing, cleanup, or transformation.

Source

Every validation failure that reaches review becomes a fixture: the raw output, the expected verdict, and the reason.

Source