Shape Free-Endpoint Load With a Batcher Before the Queue Overruns Your Free Server

Description and use

A batcher that stores tasks in a durable queue and sends them in bounded batches to a free model endpoint, preventing overload and retry amplification on a stateless server.

Problem

A stateless free server running a PR‑summary service was overwhelmed by a burst of tasks, causing 429/503 errors from a free model endpoint and stalling the worker pool.

Outcome

The batcher reduced endpoint overload, eliminated retry amplification, and allowed the stateless server to process tasks reliably without memory bloat.

Workflow

• 1. PR webhook writes a task file to an external queue directory.
• 2. A polling worker drains up to MAX_BATCH_SIZE tasks or until WINDOW_SECONDS elapse.
• 3. The drained tasks are sent as a single batch to the free model endpoint.
• 4. Results are written to a results directory keyed by PR number.
• 5. The worker sleeps MIN_INTERVAL_SECONDS before next batch.

Evidence

The first morning, the queue reached 600 tasks, the server stayed stateless, and every task that failed was re‑queued by a retry loop.

Source

The batcher has three knobs: max_batch_size, window_seconds, and min_interval_between_batches.

Source