The Model Changed Underneath Me: Pinning LLM Versions and Regression Tests in n8n

说明与用途

在 n8n 中固定 LLM 模型版本并运行回归测试可防止输出漂移，确保质量和成本控制，并简化回滚。

问题

LLM workflows in n8n can silently change output when the provider updates a model alias, causing failures that are hard to diagnose.

结果

Pinning model versions and running regression tests prevents silent drift, enables quick diagnosis of output changes, and allows controlled promotion of new model versions while monitoring cost and latency.

工作流

• 1. Pin the exact model version instead of an alias.
• 2. Record the served model string in the job row.
• 3. Maintain a small regression set of real inputs with invariant assertions.
• 4. Run the regression set against the current pinned version and the candidate version.
• 5. Diff outputs, check invariants, and compare cost/latency.
• 6. Promote only if all checks pass and keep the old version available for rollback.
• 7. Store prompt and model versions together to enable full rollback.

证据

The model alias I was calling had moved to a newer version over the weekend.

来源

Pin the exact model version, not the family name.

来源

A regression set, not a vibe check.

来源

Keep the previous pinned version working until the new one has run in production for a while.

来源

My regression run records tokens in, tokens out, and wall-clock time per input, and I compare those against the current version before promoting.

来源