Getting only the new DOIs from the Crossref API every day: dates, cursors and rate limits

Description and use

An automated daily workflow that fetches new DOIs from Crossref using the created-date filter, cursor paging, rate‑limit compliance, and deduplication, outputting results to CSV.

Problem

Need a daily CSV of journal articles that received a DOI since yesterday, without duplicates.

Outcome

The script produces a daily CSV of new journal article DOIs, handling pagination, rate limits, and deduplication, and runs automatically via GitHub Actions.

Workflow

• 1. Choose created-date filter to capture DOI registration time.
• 2. Set both from- and until- dates (inclusive) to target a specific day or window.
• 3. Use cursor paging: start with cursor='*', read next-cursor, repeat until empty.
• 4. Include sort=created&order=asc to retrieve oldest first for capped feeds.
• 5. Set rows=1000 (max) and use select to limit fields.
• 6. Respect Crossref rate limits: 1 req/s public, 3 req/s polite, honor Retry-After and back‑off.
• 7. Identify polite pool by adding mailto=your@email and User‑Agent with tool name and repo URL.
• 8. Handle late arrivals by overlapping window (yesterday + today) and dedupe by DOI.
• 9. Normalize ISSNs, DOIs, dates, titles, and handle duplicates within and across runs.
• 10. Store results in CSV with fields DOI, type, title, journal, issn, publisher, member, published, created, url, first_seen.

Evidence

I wanted a boring thing: every morning, a CSV of the journal articles that got a DOI since yesterday, for a handful of journals, and never the same DOI twice.

Source

/works has four families of date filters, and for "what is new" they give very different answers.

Source

Use deep paging: 1) Send cursor=* with the first request. 2) Read message.next-cursor from the answer and send it back with the next request. 3) Stop when a page comes back empty.

Source

Crossref changed its limits in July 2026. For list and filter requests like these: 1 request per second in the public pool; 3 per second in the polite pool; polite-pool limits counted per mailto email address.

Source

To get into the polite pool, add [email protected] to the query string or the User-Agent.

Source