The 5-Minute Wall: Splitting Long n8n Runs Before the Executor Kills Them

说明与用途

将长时间运行的 n8n 任务拆分为受限块，并在每个块后记录进度，可在失败后安全恢复，消除重复和数据丢失。

问题

A nightly batch job that processed an unbounded list of items in a single n8n execution failed mid-run when the list grew, causing duplicate outputs and loss of progress.

结果

By chunking the workload and checkpointing after each chunk, the workflow now reliably resumes from the last completed chunk, eliminating duplicate outputs and ensuring all items are processed even after interruptions.

工作流

• 1. Chunk the input list into fixed-size batches (e.g., 25 or 50 items).
• 2. For each chunk, start a new n8n execution (often via sub-workflow call).
• 3. After a chunk completes, write a progress record indicating the chunk finished and the next start index.
• 4. On restart, read the progress record and start from the next unprocessed chunk.
• 5. Include a short pause between chunks to respect downstream rate limits.
• 6. Test by forcing a mid-batch stop, restarting, and verifying no duplicates and exact output count.

证据

A nightly batch job looked simple on paper: read a list of items, run each one through an LLM step, write the results back. It ran fine for weeks, until the list grew.

来源

Restarting it from the beginning was the obvious fix, and it was the wrong one. The first 300 items ran again, re-sent outputs that had already gone out, and then the run died again, a little further along this time.

来源

Split first, then checkpoint. First, chunk the input. Second, checkpoint after every chunk.

来源

The checkpoint store does not need to be fancy. A row in a database, a sheet, a small key-value record — anything that survives the death of the execution that wrote it.

来源