One Script, Five Signals: My Free Model Server Scorecard

Description and use

The author provides a script to evaluate free model servers using five signals: TTFT, output rate, error rate, output integrity, and consistency. This method helps identify hidden reliability issues before production deployment.

Problem

Free model servers often appear reliable but fail unpredictably in production (e.g., returning garbage or timing out), making it difficult to assess their true reliability before integration.

Outcome

The workflow provides a robust method to detect gross failures in free model servers by combining latency, throughput, error, integrity, and consistency metrics. It reveals that a single benchmark run is insufficient and that retries hide critical failure modes.

Workflow

• Define five fixed tasks: summarize, code, JSON, Q&A, and rewrite.
• Configure the script with the target OpenAI-compatible endpoint URL, API key, and model name.
• Run the script to execute three repetitions of each task (15 calls total) without retries to capture raw failures.
• Measure Time to First Token (TTFT) for perceived latency.
• Calculate output rate (words per second) after the first token.
• Record error rates including HTTP 4xx/5xx, timeouts, and exceptions.
• Verify output integrity by checking if responses are complete and parseable.
• Analyze consistency by calculating variance across the repeated runs.
• Review the resulting JSON array to identify patterns of failure or stability.

Evidence

A single benchmark run tells you almost nothing. Five signals, repeated three times, tell you a lot.

Source

Retries hide failures. I want the raw failure. So the harness records everything and retries nothing.

Source