Free Workers Don't Ship Free Disk: Cap Artifact Retention First

Description and use

On a free worker, 4,000 jobs produced ~2 MB artifacts each, filling the disk and causing write failures. The solution was to compute a retention budget, reject oversized batches pre‑flight, and prune old artifacts while preserving pinned ones.

Problem

A batch of 4,000 jobs on a free worker wrote ~2 MB of artifacts each, filling the disk and causing write failures, retries, and wasted tokens.

Outcome

Disk exhaustion was prevented by calculating retention budgets, rejecting oversized batches early, and pruning old artifacts, eliminating retries and token waste.

Workflow

• 1. Observe disk full error and queue age spike.
• 2. Calculate required_bytes = queue_depth * avg_artifact_bytes * safety_factor + reserve_bytes.
• 3. Run admit.py to reject batches that exceed disk capacity.
• 4. Verify disk stats with df and file size calculations before enqueueing.
• 5. Use retain.py in dry‑run mode to identify and delete old artifacts, preserving pinned ones.
• 6. Apply deletions with --apply when safe.
• 7. Monitor disk usage and queue metrics to confirm stability.

Evidence

The batch had 4,000 jobs. Each accepted job wrote about 2 MB of artifacts.

Source

Observed: OSError: [Errno 28] No space left on device, then a growing retry count.

Source

p95 queue age crossed 15 minutes while model latency stayed flat.

Source

required_bytes = queue_depth * avg_artifact_bytes * safety_factor + reserve_bytes reserve_bytes = total_disk_bytes * reserve_ratio

Source

Exit code 75 tells your scheduler this is retryable-later, not a bug.

Source