The 5-Minute Wall: Splitting Long n8n Runs Before the Executor Kills Them

Description and use

Splitting long n8n runs into bounded chunks and checkpointing after each chunk enables safe resumption after failures, eliminating duplicates and data loss.

Problem

A nightly batch job that processed an unbounded list of items in a single n8n execution failed mid-run when the list grew, causing duplicate outputs and loss of progress.

Outcome

By chunking the workload and checkpointing after each chunk, the workflow now reliably resumes from the last completed chunk, eliminating duplicate outputs and ensuring all items are processed even after interruptions.

Workflow

• 1. Chunk the input list into fixed-size batches (e.g., 25 or 50 items).
• 2. For each chunk, start a new n8n execution (often via sub-workflow call).
• 3. After a chunk completes, write a progress record indicating the chunk finished and the next start index.
• 4. On restart, read the progress record and start from the next unprocessed chunk.
• 5. Include a short pause between chunks to respect downstream rate limits.
• 6. Test by forcing a mid-batch stop, restarting, and verifying no duplicates and exact output count.

Evidence

A nightly batch job looked simple on paper: read a list of items, run each one through an LLM step, write the results back. It ran fine for weeks, until the list grew.

Source

Restarting it from the beginning was the obvious fix, and it was the wrong one. The first 300 items ran again, re-sent outputs that had already gone out, and then the run died again, a little further along this time.

Source

Split first, then checkpoint. First, chunk the input. Second, checkpoint after every chunk.

Source

The checkpoint store does not need to be fancy. A row in a database, a sheet, a small key-value record — anything that survives the death of the execution that wrote it.

Source