I built a tool that grades the benchmark instead of the model

说明与用途

Crucible 是一个用于 LLM 基准任务的确定性评分工具，使用断言而非模型，计算六个质量因子，并提供可追溯的哈希链以实现审计。

问题

Evaluations that ask a model whether an answer is "good enough" do not provide a true benchmark; they rely on a second model’s opinion and cannot be replayed.

结果

Crucible enables reproducible, deterministic grading of benchmark tasks, revealing issues such as low discrimination, unreproducibility of closed models, and cross-language determinism bugs, and provides a tamper‑evident hash chain for auditability.

工作流

• Define assertions for a task
• Record model runs
• Apply deterministic grader to each run
• Compute six factors (determinism, discrimination, fixture seal, assertion specificity, reproduction, cost fit) weighted to 1
• Score the task 0‑100 + band + weakest factor
• Publish or fix based on score

证据

If your eval asks a language model whether an answer is "good enough", you do not have a benchmark. You have a second model with an opinion, and you cannot replay it.

来源

The task is the thing that is broken, and almost nobody is measuring it.

来源

Every eval answers "how did the model do?" Crucible also answers "is this task even worth publishing?" — with six factors, every one computed from a measured quantity.

来源