How I solved the "crashed worker" problem in Go (Lease Fencing)

说明与用途

在 Relay（Go 编写的 webhook 交付服务）中，使用 lease fencing（增量所有权令牌）来防止崩溃的工作进程写入过时状态。

问题

Background workers may crash while a task is in flight, leading to uncertain side effects and stale state updates.

结果

Lease fencing ensures that only the current owner can modify job state, eliminating stale worker updates while accepting at‑least‑once delivery duplication.

工作流

• 1. Worker claims a job and receives a lease_token (e.g., 1).
• 2. Worker stores the token in memory.
• 3. If the lease expires, another worker claims the job and the database atomically increments the token (e.g., to 2).
• 4. When the original worker resumes, it attempts to update the job using its stored token.
• 5. The database rejects the update because the token no longer matches, preventing stale writes.

证据

A lease expiration doesn’t physically stop a worker. It only means the system is allowed to give the job to someone else.

来源

Instead of treating a lease as a simple boolean lock, the database assigns an incrementing integer called a fencing token every time a job is claimed.

来源

The database only accepts the update if the worker’s token still matches the current token.

来源