Frontier Model Performance Benchmark Runner

Описание и применение

Автоматизированный скрипт на Python позволяет сравнивать фронтир‑модели Oxlo.ai по задачам логики, кода и перевода, используя локальный судью для оценки результатов.

Задача

Choosing the right flagship LLM for a product requires reproducible, task‑specific performance data rather than leaderboard screenshots.

Результат

The script produces a table of scores per model per task, enabling quick identification of the best performer for each use case.

Рабочий процесс

• Configure Oxlo.ai client and select models
• Define benchmark tasks (reasoning, code generation, translation)
• Collect answers from each model
• Define judge system prompt
• Score each answer with the judge
• Print a summary table
• Run the script and interpret results

Доказательства

We loop over tasks and models, calling Oxlo.ai for each combination.

Источник

We pass each candidate answer back to Oxlo.ai alongside the original task so the judge can assess correctness.

Источник