Your dedupe key is fine. Your dedupe state is the problem.

Описание и применение

Бот, публикующий статьи на нескольких сайтах, создавал дубликаты из‑за локального хранения состояния dedupe. Решение включало изменение ключа на (webhook|url) и ограничение очереди одной машиной, а также план по переносу состояния в общую базу данных.

Задача

The bot announced duplicate articles because dedupe state was stored locally and not shared across machines, causing idempotency to fail.

Результат

The bot now correctly handles per‑account deduplication and prevents duplicate posts across machines by enforcing a single‑machine queueing guard and planning to move state to a shared store.

Рабочий процесс

• 1. Collect today's article URLs.
• 2. Filter URLs against posted.json and queue.json using a composite key (webhook|url).
• 3. Append unique entries to queue.json.
• 4. Process queue: send entries whose scheduled time has arrived and move them to posted.json.
• 5. If running on multiple machines, enforce a single‑machine queueing guard via an environment variable; fail closed if unset.

Доказательства

Both machines evaluated their own records correctly. Both reached a correct conclusion. The announcement went out twice, and posts cannot be unsent.

Источник

The real fix is moving that state somewhere shared: Firestore, object storage, a shared filesystem.

Источник