The List That Looked Complete: Paginating Provider APIs in n8n Without Silently Dropping Records

Description and use

An n8n workflow that fetched only the first page of a paginated API silently dropped records, causing missing orders. The fix involved reading the pagination contract, looping on the provider’s cursor, deduplicating by natural ID, checkpointing progress, and setting a page cap.

Problem

A workflow that fetched only the first page of a paginated API silently dropped subsequent records, causing missing orders in production.

Outcome

The workflow now reliably processes all pages, deduplicates records, resumes after interruptions, and alerts on runaway loops, eliminating missing orders.

Workflow

• Read pagination contract (page size, cursor field, end signal).
• Loop: fetch page, record cursor, fetch next page using cursor, stop when provider signals no next page.
• Deduplicate records by natural ID during merge.
• Checkpoint progress (last cursor, last record, count) after each page.
• Enforce a maximum page count with alert.
• Reconcile final count against provider’s total if available.

Evidence

The workflow had processed exactly one page and reported success, because nobody had told it that one page was not the whole list.

Source

Rule 2: loop on the cursor, not on a page count

Source

Rule 3: dedup across pages, because pages overlap

Source

Rule 4: make the listing resumable, not restartable

Source

Rule 5: bound the list, or the list will bound you

Source