Make Pass Rate a Quorumed Histogram Before Late Workers Flip the Gate

Description and use

Problem: mutable pass rate in batch evaluation can be altered by late or duplicate scores. Solution: use immutable histogram log, epoch fencing, artifact hashing, and close histogram only when quorum is met and no in‑flight epochs remain.

Problem

A batch evaluator stored pass rate as a mutable scalar, allowing late or stale worker scores to alter the gate decision, leading to unstable model releases.

Outcome

The new protocol prevents late or stale scores from altering the pass rate after the gate opens, ensuring stable model releases and correct quorum enforcement.

Workflow

• 1. Issue a lease with (run_id, epoch, digest) to a worker.
• 2. Worker hashes artifact before scoring and sends score with epoch and digest.
• 3. Coordinator appends score to histogram log if epoch matches fenced epoch and digest matches.
• 4. Coordinator tracks accepted count and watermark; when accepted >= quorum and no in-flight epochs remain, close histogram.
• 5. After closure, any new score is rejected and logged.
• 6. Ship gate reads pass rate only from closed histogram, never from mutable scalar.

Evidence

The invariant is that a ship/no-ship pass rate is legal only when it is derived from a closed histogram of scores whose (run_id, artifact_digest) pairs are unique, whose worker epochs are still fenced in, and whose accepted count meets a declared quorum denominator.

Source

Append bins, never mutate a mean. The log stores counts per bin. The mean is a pure function of a closed snapshot.

Source