Shape Free-Endpoint Load With a Batcher Before the Queue Overruns Your Free Server

Описание и применение

Батчер, сохраняющий задачи в файловой системе, сглаживает нагрузку на бесплатный API, предотвращая 429/503 ошибки и сбои воркера.

Задача

A stateless free server running a PR‑summary service was overwhelmed by a burst of tasks, causing 429/503 errors from a free model endpoint and stalling the worker pool.

Результат

The batcher reduced endpoint overload, eliminated retry amplification, and allowed the stateless server to process tasks reliably without memory bloat.

Рабочий процесс

• 1. PR webhook writes a task file to an external queue directory.
• 2. A polling worker drains up to MAX_BATCH_SIZE tasks or until WINDOW_SECONDS elapse.
• 3. The drained tasks are sent as a single batch to the free model endpoint.
• 4. Results are written to a results directory keyed by PR number.
• 5. The worker sleeps MIN_INTERVAL_SECONDS before next batch.

Доказательства

The first morning, the queue reached 600 tasks, the server stayed stateless, and every task that failed was re‑queued by a retry loop.

Источник

The batcher has three knobs: max_batch_size, window_seconds, and min_interval_between_batches.

Источник