Shape Free-Endpoint Load With a Batcher Before the Queue Overruns Your Free Server

说明与用途

使用持久化队列和批处理的方案，避免在无状态服务器上因免费模型端点的速率限制导致的429/503错误和任务堵塞。

问题

A stateless free server running a PR‑summary service was overwhelmed by a burst of tasks, causing 429/503 errors from a free model endpoint and stalling the worker pool.

结果

The batcher reduced endpoint overload, eliminated retry amplification, and allowed the stateless server to process tasks reliably without memory bloat.

工作流

• 1. PR webhook writes a task file to an external queue directory.
• 2. A polling worker drains up to MAX_BATCH_SIZE tasks or until WINDOW_SECONDS elapse.
• 3. The drained tasks are sent as a single batch to the free model endpoint.
• 4. Results are written to a results directory keyed by PR number.
• 5. The worker sleeps MIN_INTERVAL_SECONDS before next batch.

证据

The first morning, the queue reached 600 tasks, the server stayed stateless, and every task that failed was re‑queued by a retry loop.

来源

The batcher has three knobs: max_batch_size, window_seconds, and min_interval_between_batches.

来源