Pin the Dataset, Then Grade the Grader

说明与用途

使用 SHA‑256 固定数据集和评分标准，并预先注册评审者一致性，以提高基准测试的可复现性和透明度。

问题

Benchmark scores can shift when the underlying dataset or rubric changes, yet these changes are rarely reported, making it hard to interpret score improvements.

结果

By pinning the dataset and rubric via hashes and enforcing a pre‑registered agreement floor, any drift is treated as a failed run, improving reproducibility and transparency of benchmark scores.

工作流

• Hash the tasks.jsonl, rubric.md, and prompts/system.txt with sha256sum and record the hashes in prereg.hashes.
• Create a pre‑registration JSON file that includes the experiment name, frozen_at timestamp, dataset and rubric paths with their hashes, grader names, min_kappa, slices, exclusions, and metric description.
• Run the provided Python harness before scoring; it verifies the current hashes match those in the pre‑registration file and computes Cohen’s kappa between graders.
• If the hashes differ, the harness exits with code 2; if kappa < min_kappa, it exits with code 3; otherwise it proceeds to score the model.
• Report scores per slice and keep the denominator stable by recording ambiguities instead of deleting them.

证据

Hashing the task file, the rubric, and the judge prompt costs one command and removes a whole class of accidental retreat.

来源

Pre-registration gets dismissed as academic ceremony, but its practical job here is narrow and useful: it makes the re‑run decision someone else’s to audit.

来源

The following script uses only the Python standard library, makes zero network calls, and exits non‑zero the moment a frozen input changes.

来源