Measure the Endpoint Before You Measure the Model

Описание и применение

Краткая проверка перед оценкой модели выявляет скрытые отклонения в работе эндпоинта и хранилища, обеспечивая чистую оценку модели.

Задача

Benchmark scores from hosted endpoints conflate model performance with endpoint and harness behavior, making it hard to isolate model quality.

Результат

The suite identifies silent failures—sampling drift, truncation, schema mismatch, and silent routing—before full evaluation, ensuring reported scores reflect only model behavior.

Рабочий процесс

• 1. Compute SHA‑256 of probe_set.jsonl to detect changes.
• 2. Run probe_endpoint.py with --prompts probe_set.jsonl --out report.json.
• 3. Probe_repeat: send the same deterministic prompt 5 times, hash outputs, compute agreement.
• 4. Probe_needle: embed a unique canary string in a long synthetic document to detect truncation.
• 5. Probe_schema: force a tool call with a required enum field to detect schema drift.
• 6. Collect metrics: distinct outputs, agreement, served_models, latency.
• 7. Apply reporting rule: if any probe fails, reject the score for publication.

Доказательства

A benchmark score collected through a hosted endpoint measures three things at once, and only one of them is the model.

Источник

The suite reads a JSONL file of probe prompts and records its SHA‑256 before the run, so a later reader can tell whether the suite or the endpoint changed.

Источник

probe_repeat: send the same deterministic prompt 5 times, hash outputs, compute agreement.

Источник