Measure the Endpoint Before You Measure the Model

Description and use

A brief pre‑evaluation suite isolates endpoint and harness drift, yielding a clean model benchmark.

Problem

Benchmark scores from hosted endpoints conflate model performance with endpoint and harness behavior, making it hard to isolate model quality.

Outcome

The suite identifies silent failures—sampling drift, truncation, schema mismatch, and silent routing—before full evaluation, ensuring reported scores reflect only model behavior.

Workflow

• 1. Compute SHA‑256 of probe_set.jsonl to detect changes.
• 2. Run probe_endpoint.py with --prompts probe_set.jsonl --out report.json.
• 3. Probe_repeat: send the same deterministic prompt 5 times, hash outputs, compute agreement.
• 4. Probe_needle: embed a unique canary string in a long synthetic document to detect truncation.
• 5. Probe_schema: force a tool call with a required enum field to detect schema drift.
• 6. Collect metrics: distinct outputs, agreement, served_models, latency.
• 7. Apply reporting rule: if any probe fails, reject the score for publication.

Evidence

A benchmark score collected through a hosted endpoint measures three things at once, and only one of them is the model.

Source

The suite reads a JSONL file of probe prompts and records its SHA‑256 before the run, so a later reader can tell whether the suite or the endpoint changed.

Source

probe_repeat: send the same deterministic prompt 5 times, hash outputs, compute agreement.

Source