8 Chinese AI models on the same Pi coding-agent task: what we measured

Описание и применение

Сравнение восьми китайских ИИ‑моделей на одной задаче с использованием агента Pi показало, что все модели успешно завершили задачу, предоставив прозрачные метрики запросов, токенов и стоимости.

Задача

Determine how eight Chinese AI models perform on a single coding task when accessed through the same Pi agent, measuring success, request count, token usage, and cost.

Результат

All eight models succeeded on the task; the benchmark provides transparent metrics on requests, tokens, and cost, enabling operational decision‑making.

Рабочий процесс

• 1. Prepare identical fixture and test directory for each model.
• 2. Configure Pi agent with the same provider settings and task contract.
• 3. Run the isolated JavaScript coding task for each model, ensuring separate work directories.
• 4. Record Pi requests, token usage, and production‑audited cost per run.
• 5. Store raw run evidence separately from the task workspace.
• 6. Compile results into a model‑by‑model table and downloadable JSON.
• 7. Publish the benchmark page with methodology notes and data.

Доказательства

All eight models completed the task successfully.

Источник

45 Pi requests, 94,502 tokens, $0.037618 production‑audited billed cost

Источник