The List That Looked Complete: Paginating Provider APIs in n8n Without Silently Dropping Records

Описание и применение

В n8n‑workflow, который извлекал только первую страницу API, пропускались записи, что приводило к отсутствию заказов. Решение включало чтение контракта пагинации, цикл по курсору, дедупликацию по ID, контроль прогресса и ограничение количества страниц.

Задача

A workflow that fetched only the first page of a paginated API silently dropped subsequent records, causing missing orders in production.

Результат

The workflow now reliably processes all pages, deduplicates records, resumes after interruptions, and alerts on runaway loops, eliminating missing orders.

Рабочий процесс

• Read pagination contract (page size, cursor field, end signal).
• Loop: fetch page, record cursor, fetch next page using cursor, stop when provider signals no next page.
• Deduplicate records by natural ID during merge.
• Checkpoint progress (last cursor, last record, count) after each page.
• Enforce a maximum page count with alert.
• Reconcile final count against provider’s total if available.

Доказательства

The workflow had processed exactly one page and reported success, because nobody had told it that one page was not the whole list.

Источник

Rule 2: loop on the cursor, not on a page count

Источник

Rule 3: dedup across pages, because pages overlap

Источник

Rule 4: make the listing resumable, not restartable

Источник

Rule 5: bound the list, or the list will bound you

Источник