How I Built a Substack New-Post Monitor (and the Dedup Problem Nobody Warns You About)

说明与用途

构建了一个 Substack 监控工具，使用稳定的帖子 ID 并原子地保存状态，以避免重复警报。

问题

Prevent duplicate alerts for the same Substack post across runs, crashes, and machine moves.

结果

The actor reliably suppresses duplicate alerts, recovers from crashes, and isolates state per publication, costing only a fraction of a cent per scheduled check.

工作流

• Fetch Substack archive API for the publication
• Parse feed entries and extract stable post IDs
• Load persisted seen IDs from shared key‑value store
• Compute set difference to find new posts
• Persist updated seen IDs and pending‑alerts queue atomically
• Drain any pending alerts queue and send alerts
• Store new posts in output dataset

证据

The finished actor does: You give it a publication URL and a schedule. It fetches via Substack's public archive API, dedups on Substack's stable post IDs, persists its snapshot in a named key-value store, and emits only genuinely new posts to the output dataset.

来源

The lesson: the state file is the product, and it needs to live where the compute lives.

来源

Persist seen IDs and a pending‑alerts queue atomically, then drain the queue.

来源