DeepSeek's own table shows the same model dropping 90.6 84.1 across five harnesses

Description and use

Benchmark scores for a single model vary widely depending on the harness used, revealing that model performance is inseparable from its scaffold.

Problem

Benchmark scores for a single model vary widely depending on the harness used, obscuring true model performance.

Outcome

The harness around a model can shift benchmark scores by up to a third of the scale, demonstrating that model performance is inseparable from its scaffold.

Workflow

• 1. Select a single model (DeepSeek V4.1‑Flash).
• 2. Run the model on Terminal‑Bench 2.1 under each of five harnesses: DeepSeek Harness (Minimal), Claude Code, Pi, OpenCode, Codex.
• 3. Record the resulting scores: 90.6, 88.0, 86.1, 85.0, 84.1 respectively.
• 4. Perform independent neutral‑harness runs (e.g., Vals AI) to obtain baseline scores (74.53%).
• 5. Compare vendor‑reported scores with neutral‑harness results to quantify harness impact.
• 6. Publish full evaluation tables, including harness names and raw outputs, to enable transparent assessment.

Evidence

Same weights. Same prompts, same tasks. A 6.5-point spread decided entirely by the scaffolding the model runs inside.

Source

When an independent tester, Vals AI, ran the model on Terminal‑Bench 2.1 under neutral conditions, they measured 74.53% — against the 90.6 vendor number.

Source

The same pattern showed up on the previous flagship: DeepSeek reported 87.9 on Terminal‑Bench 2.1 for V4‑Pro 0813; Vals AI measured 54.68% on the reference harness — a 33-point gap.

Source

Emma Intelligence started as exactly that bet: the bottleneck isn't the model, it's the scaffold around it.

Source