Validating LLM Output in n8n

Описание и применение

В n8n‑потоке LLM возвращал корректный JSON, но с неверными значениями, что приводило к ошибкам маршрутизации. Внедрена двухступенчатая проверка: строгий схематический валидатор и семантические проверки (список допустимых категорий, диапазон приоритетов, существующие ID). Ошибки направляются в очередь проверки, а не повторяются. Логи исходного вывода сохраняются, а реальные ошибки становятся фикстурами для регрессии.

Задача

An n8n workflow that classifies incoming messages via an LLM returned valid JSON that was semantically incorrect, causing mis‑routing and priority errors.

Результат

The workflow now reliably rejects semantically incorrect JSON, preventing mis‑routing and priority errors while maintaining fast deterministic validation and a review process for corrections.

Рабочий процесс

• 1. Send message to LLM and request JSON output
• 2. Parse JSON with strict schema validation
• 3. Perform meaning checks on each field (category, priority, dates, IDs)
• 4. If any check fails, route to review queue with raw output, errors, and original input
• 5. For format‑only failures, run a deterministic cleanup pass before review
• 6. Log raw LLM output on every run
• 7. Add each real failure to regression fixture set
• 8. Run fixtures on every validation change to catch drift

Доказательства

The JSON node never complained. The workflow never errored. And once or twice a week, a message landed in the wrong queue, or with a priority that made no sense, or with a summary describing a different message entirely.

Источник

Parsing is not validation. If your workflow treats "it parsed" as "it is correct", you have no validation at all — you have a parser with good manners.

Источник

When validation fails, the tempting move is to retry the model call. ... a model that invents a category will often invent it again, slightly differently, and each retry is another chance for a plausible‑looking wrong answer to slip through.

Источник

The most useful debugging artifact in these workflows is the raw model output, before any parsing, cleanup, or transformation.

Источник

Every validation failure that reaches review becomes a fixture: the raw output, the expected verdict, and the reason.

Источник