I built a tool that grades the benchmark instead of the model

Description and use

Crucible is a deterministic grader for LLM benchmark tasks that uses assertions instead of models, computes six quality factors, and provides a tamper‑evident hash chain for auditability.

Problem

Evaluations that ask a model whether an answer is "good enough" do not provide a true benchmark; they rely on a second model’s opinion and cannot be replayed.

Outcome

Crucible enables reproducible, deterministic grading of benchmark tasks, revealing issues such as low discrimination, unreproducibility of closed models, and cross-language determinism bugs, and provides a tamper‑evident hash chain for auditability.

Workflow

• Define assertions for a task
• Record model runs
• Apply deterministic grader to each run
• Compute six factors (determinism, discrimination, fixture seal, assertion specificity, reproduction, cost fit) weighted to 1
• Score the task 0‑100 + band + weakest factor
• Publish or fix based on score

Evidence

If your eval asks a language model whether an answer is "good enough", you do not have a benchmark. You have a second model with an opinion, and you cannot replay it.

Source

The task is the thing that is broken, and almost nobody is measuring it.

Source

Every eval answers "how did the model do?" Crucible also answers "is this task even worth publishing?" — with six factors, every one computed from a measured quantity.

Source