Pin the Dataset, Then Grade the Grader

Description and use

Pinning the dataset and rubric with SHA‑256 and pre‑registering grader agreement improves reproducibility and transparency of benchmark scores.

Problem

Benchmark scores can shift when the underlying dataset or rubric changes, yet these changes are rarely reported, making it hard to interpret score improvements.

Outcome

By pinning the dataset and rubric via hashes and enforcing a pre‑registered agreement floor, any drift is treated as a failed run, improving reproducibility and transparency of benchmark scores.

Workflow

• Hash the tasks.jsonl, rubric.md, and prompts/system.txt with sha256sum and record the hashes in prereg.hashes.
• Create a pre‑registration JSON file that includes the experiment name, frozen_at timestamp, dataset and rubric paths with their hashes, grader names, min_kappa, slices, exclusions, and metric description.
• Run the provided Python harness before scoring; it verifies the current hashes match those in the pre‑registration file and computes Cohen’s kappa between graders.
• If the hashes differ, the harness exits with code 2; if kappa < min_kappa, it exits with code 3; otherwise it proceeds to score the model.
• Report scores per slice and keep the denominator stable by recording ambiguities instead of deleting them.

Evidence

Hashing the task file, the rubric, and the judge prompt costs one command and removes a whole class of accidental retreat.

Source

Pre-registration gets dismissed as academic ceremony, but its practical job here is narrow and useful: it makes the re‑run decision someone else’s to audit.

Source

The following script uses only the Python standard library, makes zero network calls, and exits non‑zero the moment a frozen input changes.

Source