I built a tool that grades the benchmark instead of the model

Описание и применение

Crucible — инструмент для детерминированной оценки задач бенчмарков LLM, который использует утверждения вместо моделей, вычисляет шесть факторов качества и обеспечивает хэш‑цепочку для аудита.

Задача

Evaluations that ask a model whether an answer is "good enough" do not provide a true benchmark; they rely on a second model’s opinion and cannot be replayed.

Результат

Crucible enables reproducible, deterministic grading of benchmark tasks, revealing issues such as low discrimination, unreproducibility of closed models, and cross-language determinism bugs, and provides a tamper‑evident hash chain for auditability.

Рабочий процесс

• Define assertions for a task
• Record model runs
• Apply deterministic grader to each run
• Compute six factors (determinism, discrimination, fixture seal, assertion specificity, reproduction, cost fit) weighted to 1
• Score the task 0‑100 + band + weakest factor
• Publish or fix based on score

Доказательства

If your eval asks a language model whether an answer is "good enough", you do not have a benchmark. You have a second model with an opinion, and you cannot replay it.

Источник

The task is the thing that is broken, and almost nobody is measuring it.

Источник

Every eval answers "how did the model do?" Crucible also answers "is this task even worth publishing?" — with six factors, every one computed from a measured quantity.

Источник