The List That Looked Complete: Paginating Provider APIs in n8n Without Silently Dropping Records

说明与用途

n8n 工作流仅获取分页 API 的第一页，导致记录被忽略，订单缺失。解决方案包括读取分页协议、按游标循环、按自然 ID 去重、进度检查点以及设置页面上限。

问题

A workflow that fetched only the first page of a paginated API silently dropped subsequent records, causing missing orders in production.

结果

The workflow now reliably processes all pages, deduplicates records, resumes after interruptions, and alerts on runaway loops, eliminating missing orders.

工作流

• Read pagination contract (page size, cursor field, end signal).
• Loop: fetch page, record cursor, fetch next page using cursor, stop when provider signals no next page.
• Deduplicate records by natural ID during merge.
• Checkpoint progress (last cursor, last record, count) after each page.
• Enforce a maximum page count with alert.
• Reconcile final count against provider’s total if available.

证据

The workflow had processed exactly one page and reported success, because nobody had told it that one page was not the whole list.

来源

Rule 2: loop on the cursor, not on a page count

来源

Rule 3: dedup across pages, because pages overlap

来源

Rule 4: make the listing resumable, not restartable

来源

Rule 5: bound the list, or the list will bound you

来源