How I stopped losing 3 hours of GPU work every time a job crashed

Description and use

Implementing atomic checkpoints after each stage and a resume logic prevented full restarts of GPU‑heavy jobs after crashes, saving hours of computation.

Problem

GPU‑heavy jobs crashed mid‑run, forcing a full restart and wasting hours of expensive GPU time.

Outcome

After implementing atomic stage checkpoints and resume logic, a crash at 95% no longer caused a full restart; instead the pipeline resumed from the last completed stage, saving GPU time and reducing manual intervention.

Workflow

• After each stage, write a "[done]" flag to a state file atomically (tmp + rename).
• Implement run() to execute pending stages and resume() to skip done stages and retry failed ones.
• Ensure stages are idempotent or own local temp dirs to avoid duplicate side effects.
• Add worker heartbeats and a watchdog to detect hung workers via staleness checks.
• Test crash scenarios to verify resume behavior and atomicity.

Evidence

If the process died in render, we started over. Every time.

Source

After every stage, write down that you finished it.

Source

state is written atomically (tmp file → rename), so a kill mid-write doesn't corrupt everything

Source

workers send a heartbeat; watchdog notices when they go silent

Source

Long jobs: tens of minutes to hours; Before: crash = full re-run; After: crash = re-run from last finished stage

Source