How I solved the "crashed worker" problem in Go (Lease Fencing)

Description and use

In Relay, a Go webhook delivery service, lease fencing with an incrementing ownership token prevents stale workers from overwriting newer state after a crash.

Problem

Background workers may crash while a task is in flight, leading to uncertain side effects and stale state updates.

Outcome

Lease fencing ensures that only the current owner can modify job state, eliminating stale worker updates while accepting at‑least‑once delivery duplication.

Workflow

• 1. Worker claims a job and receives a lease_token (e.g., 1).
• 2. Worker stores the token in memory.
• 3. If the lease expires, another worker claims the job and the database atomically increments the token (e.g., to 2).
• 4. When the original worker resumes, it attempts to update the job using its stored token.
• 5. The database rejects the update because the token no longer matches, preventing stale writes.

Evidence

A lease expiration doesn’t physically stop a worker. It only means the system is allowed to give the job to someone else.

Source

Instead of treating a lease as a simple boolean lock, the database assigns an incrementing integer called a fencing token every time a job is claimed.

Source

The database only accepts the update if the worker’s token still matches the current token.

Source