Measure the Endpoint Before You Measure the Model

说明与用途

在评估前运行简短的合规套件，隔离端点和工具箱漂移，得到干净的模型基准。

问题

Benchmark scores from hosted endpoints conflate model performance with endpoint and harness behavior, making it hard to isolate model quality.

结果

The suite identifies silent failures—sampling drift, truncation, schema mismatch, and silent routing—before full evaluation, ensuring reported scores reflect only model behavior.

工作流

• 1. Compute SHA‑256 of probe_set.jsonl to detect changes.
• 2. Run probe_endpoint.py with --prompts probe_set.jsonl --out report.json.
• 3. Probe_repeat: send the same deterministic prompt 5 times, hash outputs, compute agreement.
• 4. Probe_needle: embed a unique canary string in a long synthetic document to detect truncation.
• 5. Probe_schema: force a tool call with a required enum field to detect schema drift.
• 6. Collect metrics: distinct outputs, agreement, served_models, latency.
• 7. Apply reporting rule: if any probe fails, reject the score for publication.

证据

A benchmark score collected through a hosted endpoint measures three things at once, and only one of them is the model.

来源

The suite reads a JSONL file of probe prompts and records its SHA‑256 before the run, so a later reader can tell whether the suite or the endpoint changed.

来源

probe_repeat: send the same deterministic prompt 5 times, hash outputs, compute agreement.

来源