How I stopped losing 3 hours of GPU work every time a job crashed

说明与用途

通过在每个阶段后实现原子化检查点和恢复逻辑，避免了GPU密集型任务在崩溃后完全重新启动，从而节省了数小时的计算时间。

问题

GPU‑heavy jobs crashed mid‑run, forcing a full restart and wasting hours of expensive GPU time.

结果

After implementing atomic stage checkpoints and resume logic, a crash at 95% no longer caused a full restart; instead the pipeline resumed from the last completed stage, saving GPU time and reducing manual intervention.

工作流

• After each stage, write a "[done]" flag to a state file atomically (tmp + rename).
• Implement run() to execute pending stages and resume() to skip done stages and retry failed ones.
• Ensure stages are idempotent or own local temp dirs to avoid duplicate side effects.
• Add worker heartbeats and a watchdog to detect hung workers via staleness checks.
• Test crash scenarios to verify resume behavior and atomicity.

证据

If the process died in render, we started over. Every time.

来源

After every stage, write down that you finished it.

来源

state is written atomically (tmp file → rename), so a kill mid-write doesn't corrupt everything

来源

workers send a heartbeat; watchdog notices when they go silent

来源

Long jobs: tens of minutes to hours; Before: crash = full re-run; After: crash = re-run from last finished stage

来源