8 Chinese AI models on the same Pi coding-agent task: what we measured

说明与用途

使用 Pi 代理对八个中文 AI 模型进行单一编码任务的基准测试，结果显示所有模型均成功完成任务，并提供了请求数、令牌数和成本等透明指标。

问题

Determine how eight Chinese AI models perform on a single coding task when accessed through the same Pi agent, measuring success, request count, token usage, and cost.

结果

All eight models succeeded on the task; the benchmark provides transparent metrics on requests, tokens, and cost, enabling operational decision‑making.

工作流

• 1. Prepare identical fixture and test directory for each model.
• 2. Configure Pi agent with the same provider settings and task contract.
• 3. Run the isolated JavaScript coding task for each model, ensuring separate work directories.
• 4. Record Pi requests, token usage, and production‑audited cost per run.
• 5. Store raw run evidence separately from the task workspace.
• 6. Compile results into a model‑by‑model table and downloadable JSON.
• 7. Publish the benchmark page with methodology notes and data.

证据

All eight models completed the task successfully.

来源

45 Pi requests, 94,502 tokens, $0.037618 production‑audited billed cost

来源