Free Workers Don't Ship Free Disk: Cap Artifact Retention First

说明与用途

在免费工作节点上，4000个作业每个产生约2 MB的工件，导致磁盘填满并出现写入错误。解决方案是计算保留预算，预先拒绝过大的批次，并在保留已固定工件的同时清理旧工件。

问题

A batch of 4,000 jobs on a free worker wrote ~2 MB of artifacts each, filling the disk and causing write failures, retries, and wasted tokens.

结果

Disk exhaustion was prevented by calculating retention budgets, rejecting oversized batches early, and pruning old artifacts, eliminating retries and token waste.

工作流

• 1. Observe disk full error and queue age spike.
• 2. Calculate required_bytes = queue_depth * avg_artifact_bytes * safety_factor + reserve_bytes.
• 3. Run admit.py to reject batches that exceed disk capacity.
• 4. Verify disk stats with df and file size calculations before enqueueing.
• 5. Use retain.py in dry‑run mode to identify and delete old artifacts, preserving pinned ones.
• 6. Apply deletions with --apply when safe.
• 7. Monitor disk usage and queue metrics to confirm stability.

证据

The batch had 4,000 jobs. Each accepted job wrote about 2 MB of artifacts.

来源

Observed: OSError: [Errno 28] No space left on device, then a growing retry count.

来源

p95 queue age crossed 15 minutes while model latency stayed flat.

来源

required_bytes = queue_depth * avg_artifact_bytes * safety_factor + reserve_bytes reserve_bytes = total_disk_bytes * reserve_ratio

来源

Exit code 75 tells your scheduler this is retryable-later, not a bug.

来源