Free Workers Don't Ship Free Disk: Cap Artifact Retention First

Описание и применение

На бесплатном воркере 4000 задач создали ~2 МБ артефактов каждая, заполнив диск и вызывая ошибки записи. Решение: вычислить бюджет хранения, отклонять большие пакеты до запуска и удалять старые артефакты, сохраняя закрепленные.

Задача

A batch of 4,000 jobs on a free worker wrote ~2 MB of artifacts each, filling the disk and causing write failures, retries, and wasted tokens.

Результат

Disk exhaustion was prevented by calculating retention budgets, rejecting oversized batches early, and pruning old artifacts, eliminating retries and token waste.

Рабочий процесс

• 1. Observe disk full error and queue age spike.
• 2. Calculate required_bytes = queue_depth * avg_artifact_bytes * safety_factor + reserve_bytes.
• 3. Run admit.py to reject batches that exceed disk capacity.
• 4. Verify disk stats with df and file size calculations before enqueueing.
• 5. Use retain.py in dry‑run mode to identify and delete old artifacts, preserving pinned ones.
• 6. Apply deletions with --apply when safe.
• 7. Monitor disk usage and queue metrics to confirm stability.

Доказательства

The batch had 4,000 jobs. Each accepted job wrote about 2 MB of artifacts.

Источник

Observed: OSError: [Errno 28] No space left on device, then a growing retry count.

Источник

p95 queue age crossed 15 minutes while model latency stayed flat.

Источник

required_bytes = queue_depth * avg_artifact_bytes * safety_factor + reserve_bytes reserve_bytes = total_disk_bytes * reserve_ratio

Источник

Exit code 75 tells your scheduler this is retryable-later, not a bug.

Источник