How I Built a Substack New-Post Monitor (and the Dedup Problem Nobody Warns You About)

Description and use

Built a Substack monitor that suppresses duplicate alerts by using stable post IDs and atomic state persistence.

Problem

Prevent duplicate alerts for the same Substack post across runs, crashes, and machine moves.

Outcome

The actor reliably suppresses duplicate alerts, recovers from crashes, and isolates state per publication, costing only a fraction of a cent per scheduled check.

Workflow

• Fetch Substack archive API for the publication
• Parse feed entries and extract stable post IDs
• Load persisted seen IDs from shared key‑value store
• Compute set difference to find new posts
• Persist updated seen IDs and pending‑alerts queue atomically
• Drain any pending alerts queue and send alerts
• Store new posts in output dataset

Evidence

The finished actor does: You give it a publication URL and a schedule. It fetches via Substack's public archive API, dedups on Substack's stable post IDs, persists its snapshot in a named key-value store, and emits only genuinely new posts to the output dataset.

Source

The lesson: the state file is the product, and it needs to live where the compute lives.

Source

Persist seen IDs and a pending‑alerts queue atomically, then drain the queue.

Source