How I solved the "crashed worker" problem in Go (Lease Fencing)

Описание и применение

В Relay, сервисе доставки вебхуков на Go, реализован механизм lease fencing, который использует инкрементный токен владения для предотвращения записи устаревшего состояния после сбоя воркера.

Задача

Background workers may crash while a task is in flight, leading to uncertain side effects and stale state updates.

Результат

Lease fencing ensures that only the current owner can modify job state, eliminating stale worker updates while accepting at‑least‑once delivery duplication.

Рабочий процесс

• 1. Worker claims a job and receives a lease_token (e.g., 1).
• 2. Worker stores the token in memory.
• 3. If the lease expires, another worker claims the job and the database atomically increments the token (e.g., to 2).
• 4. When the original worker resumes, it attempts to update the job using its stored token.
• 5. The database rejects the update because the token no longer matches, preventing stale writes.

Доказательства

A lease expiration doesn’t physically stop a worker. It only means the system is allowed to give the job to someone else.

Источник

Instead of treating a lease as a simple boolean lock, the database assigns an incrementing integer called a fencing token every time a job is claimed.

Источник

The database only accepts the update if the worker’s token still matches the current token.

Источник