Celery visibility_timeout Ran My Claude Jobs 3.3x: 1,800 Became 5,913

Описание и применение

Celery с Redis повторно выполняет задачи, если они не подтверждены в течение visibility_timeout, что приводит к лишним вызовам Claude. Решение: увеличить visibility_timeout, не использовать длинные countdowns, добавить ключ идемпотентности.

Задача

Celery tasks scheduled with long countdowns on Redis were re‑delivered repeatedly due to the broker’s visibility_timeout, causing 3.3× more Claude API calls than intended.

Результат

After raising visibility_timeout, replacing long countdowns with just‑in‑time enqueuing, and adding an idempotency key, the number of Claude calls dropped to 1,803 (3 extra real retries) while maintaining one summary per ticket.

Рабочий процесс

• Schedule 1,800 tasks with countdown=i*12 seconds
• Run tasks on four workers; each task acknowledges on start
• Redis broker restores unacknowledged messages after visibility_timeout
• Redelivered tasks run again, each making a Claude call and upserting
• Duplicate calls cause bursts and rate‑limit errors
• Add logging to count task executions
• Analyze logs to confirm duplication pattern
• Implement fixes: raise visibility_timeout, stop long countdowns, add idempotency key

Доказательства

Celery visibility_timeout Ran My Claude Jobs 3.3x: 1,800 Became 5,913

Источник

Celery visibility_timeout on my Redis broker was quietly re‑running most of those tasks, some of them six times, and my upsert erased the evidence every time.

Источник

I spread 1,800 tasks over 6 hours with countdown=i * 12. Result: 5,913 Claude calls, 3.3x what I needed, and 1,500 of 1,800 task IDs ran more than once.

Источник

Fix: stop using multi‑hour countdowns for rate limiting, feed the queue just‑in‑time instead, and add an idempotency key, because Celery on Redis is at‑least‑once no matter what you configure.

Источник