How I stopped losing 3 hours of GPU work every time a job crashed

Описание и применение

Внедрение атомарных чекпоинтов после каждого этапа и логики возобновления позволило избежать полной перезаписи GPU‑тяжёлых задач после падения, экономя часы работы.

Задача

GPU‑heavy jobs crashed mid‑run, forcing a full restart and wasting hours of expensive GPU time.

Результат

After implementing atomic stage checkpoints and resume logic, a crash at 95% no longer caused a full restart; instead the pipeline resumed from the last completed stage, saving GPU time and reducing manual intervention.

Рабочий процесс

• After each stage, write a "[done]" flag to a state file atomically (tmp + rename).
• Implement run() to execute pending stages and resume() to skip done stages and retry failed ones.
• Ensure stages are idempotent or own local temp dirs to avoid duplicate side effects.
• Add worker heartbeats and a watchdog to detect hung workers via staleness checks.
• Test crash scenarios to verify resume behavior and atomicity.

Доказательства

If the process died in render, we started over. Every time.

Источник

After every stage, write down that you finished it.

Источник

state is written atomically (tmp file → rename), so a kill mid-write doesn't corrupt everything

Источник

workers send a heartbeat; watchdog notices when they go silent

Источник

Long jobs: tens of minutes to hours; Before: crash = full re-run; After: crash = re-run from last finished stage

Источник