DeepSeek's own table shows the same model dropping 90.6 84.1 across five harnesses

说明与用途

同一模型在不同的支架下得分差异显著，表明模型性能与其支架密不可分。

问题

Benchmark scores for a single model vary widely depending on the harness used, obscuring true model performance.

结果

The harness around a model can shift benchmark scores by up to a third of the scale, demonstrating that model performance is inseparable from its scaffold.

工作流

• 1. Select a single model (DeepSeek V4.1‑Flash).
• 2. Run the model on Terminal‑Bench 2.1 under each of five harnesses: DeepSeek Harness (Minimal), Claude Code, Pi, OpenCode, Codex.
• 3. Record the resulting scores: 90.6, 88.0, 86.1, 85.0, 84.1 respectively.
• 4. Perform independent neutral‑harness runs (e.g., Vals AI) to obtain baseline scores (74.53%).
• 5. Compare vendor‑reported scores with neutral‑harness results to quantify harness impact.
• 6. Publish full evaluation tables, including harness names and raw outputs, to enable transparent assessment.

证据

Same weights. Same prompts, same tasks. A 6.5-point spread decided entirely by the scaffolding the model runs inside.

来源

When an independent tester, Vals AI, ran the model on Terminal‑Bench 2.1 under neutral conditions, they measured 74.53% — against the 90.6 vendor number.

来源

The same pattern showed up on the previous flagship: DeepSeek reported 87.9 on Terminal‑Bench 2.1 for V4‑Pro 0813; Vals AI measured 54.68% on the reference harness — a 33-point gap.

来源

Emma Intelligence started as exactly that bet: the bottleneck isn't the model, it's the scaffold around it.

来源