Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

Don't Trust Anyone's Muse Wonder Benchmark. Race It Against Opus 5.5 on Your Repo

Описание и применение

В статье описывается пошаговый рецепт сравнения моделей Muse Wonder и Opus 5.5 на собственном репозитории, используя одинаковый запрос и изолированные git‑ветки или сессии Hermes IDE.

Задача

Meta’s published benchmarks for Muse Wonder and Claude Opus 5.5 are not directly comparable because they use different prompts, tools, and harnesses. Developers need a reproducible way to compare the two models on their own codebase.

Результат

The workflow enables side‑by‑side comparison of Muse Wonder and Opus 5.5 on the same codebase, ensuring identical prompts, harnesses, and environment, thereby producing trustworthy benchmark results.

Рабочий процесс

  • Choose a failing test ticket and a prompt that fixes it.
  • Export a Meta Model API key as MODEL_API_KEY.
  • Create a wrapper script claude-muse that sets ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN, ANTHROPIC_MODEL and related env vars to muse-wonder-1.0 and execs claude.
  • Launch Hermes IDE session A with Claude Code and Opus 5.5, paste the prompt and run.
  • Launch Hermes IDE session B with Custom agent using claude-muse, paste the same prompt and run.
  • (Optional) Use plain git worktrees: git worktree add -b bakeoff/opus ../bakeoff-opus and bakeoff/muse, then run claude and claude-muse in separate terminals.
  • (Optional) Install Meta’s muse CLI and run it in a third session for Meta’s own harness.

Доказательства

Meta launched Muse Wonder on October 12, 2026

Источник

Both cost $4 per million input tokens and $20 per million output

Источник

Meta’s footnote on its SWE-Marathon row calls that row "not a controlled common-harness comparison"

Источник

  • Дата записи: 2026-10-12T21:55:53.357Z