Celery visibility_timeout Ran My Claude Jobs 3.3x: 1,800 Became 5,913

Description and use

Celery on Redis re‑delivers tasks that stay unacknowledged beyond visibility_timeout, causing excessive Claude API calls. Fixes include raising visibility_timeout, avoiding long countdowns, and adding idempotency keys.

Problem

Celery tasks scheduled with long countdowns on Redis were re‑delivered repeatedly due to the broker’s visibility_timeout, causing 3.3× more Claude API calls than intended.

Outcome

After raising visibility_timeout, replacing long countdowns with just‑in‑time enqueuing, and adding an idempotency key, the number of Claude calls dropped to 1,803 (3 extra real retries) while maintaining one summary per ticket.

Workflow

• Schedule 1,800 tasks with countdown=i*12 seconds
• Run tasks on four workers; each task acknowledges on start
• Redis broker restores unacknowledged messages after visibility_timeout
• Redelivered tasks run again, each making a Claude call and upserting
• Duplicate calls cause bursts and rate‑limit errors
• Add logging to count task executions
• Analyze logs to confirm duplication pattern
• Implement fixes: raise visibility_timeout, stop long countdowns, add idempotency key

Evidence

Celery visibility_timeout Ran My Claude Jobs 3.3x: 1,800 Became 5,913

Source

Celery visibility_timeout on my Redis broker was quietly re‑running most of those tasks, some of them six times, and my upsert erased the evidence every time.

Source

I spread 1,800 tasks over 6 hours with countdown=i * 12. Result: 5,913 Claude calls, 3.3x what I needed, and 1,500 of 1,800 task IDs ran more than once.

Source

Fix: stop using multi‑hour countdowns for rate limiting, feed the queue just‑in‑time instead, and add an idempotency key, because Celery on Redis is at‑least‑once no matter what you configure.

Source